-- Logs begin at Tue 2022-09-13 00:39:22 UTC, end at Tue 2022-09-13 01:35:13 UTC. -- Sep 13 01:04:12.119091 dv-cinder-856941-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 13 01:04:12.127574 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 13 01:04:12.129505 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: *** Starting uWSGI 2.0.18-debian (64bit) on [Tue Sep 13 01:04:12 2022] *** Sep 13 01:04:12.129505 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 13 01:04:12.129505 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 13 01:04:12.129505 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: nodename: dv-cinder-856941-3-iscsi Sep 13 01:04:12.129505 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: machine: x86_64 Sep 13 01:04:12.129505 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: clock source: unix Sep 13 01:04:12.129505 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: pcre jit disabled Sep 13 01:04:12.129505 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: detected number of CPU cores: 4 Sep 13 01:04:12.129505 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: current working directory: / Sep 13 01:04:12.129505 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: detected binary path: /usr/bin/uwsgi-core Sep 13 01:04:12.129505 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: your processes number limit is 47680 Sep 13 01:04:12.130066 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: your memory page size is 4096 bytes Sep 13 01:04:12.130066 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: detected max file descriptor number: 2048 Sep 13 01:04:12.130066 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: lock engine: pthread robust mutexes Sep 13 01:04:12.130066 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: thunder lock: enabled Sep 13 01:04:12.130066 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 13 01:04:12.130066 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 13 01:04:12.169373 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: Python main interpreter initialized at 0x5571e23a69c0 Sep 13 01:04:12.169373 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: python threads support enabled Sep 13 01:04:12.169373 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: your server socket listen backlog is limited to 100 connections Sep 13 01:04:12.169373 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: your mercy for graceful operations on workers is 90 seconds Sep 13 01:04:12.169699 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: mapped 403077 bytes (393 KB) for 2 cores Sep 13 01:04:12.169774 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: *** Operational MODE: preforking *** Sep 13 01:04:12.170042 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: *** uWSGI is running in multiple interpreter mode *** Sep 13 01:04:12.170042 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: spawned uWSGI master process (pid: 81393) Sep 13 01:04:12.170229 dv-cinder-856941-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 13 01:04:12.171593 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: spawned uWSGI worker 1 (pid: 81404, cores: 1) Sep 13 01:04:12.171593 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: spawned uWSGI worker 2 (pid: 81405, cores: 1) Sep 13 01:04:12.171593 dv-cinder-856941-3-iscsi devstack@keystone.service[81393]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 13 01:04:16.442978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 13 01:04:16.442978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: from cryptography.utils import int_from_bytes Sep 13 01:04:16.443675 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 13 01:04:16.443675 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: from cryptography.utils import int_from_bytes Sep 13 01:04:16.443820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 13 01:04:16.443820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: from cryptography.utils import int_from_bytes Sep 13 01:04:16.443820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 13 01:04:16.443820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: from cryptography.utils import int_from_bytes Sep 13 01:04:16.493238 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 13 01:04:16.493394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 13 01:04:16.493489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 13 01:04:16.493588 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 13 01:04:16.493667 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] command line args: [] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 13 01:04:16.493740 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] command line args: [] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 13 01:04:16.493891 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 13 01:04:16.494001 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 13 01:04:16.494092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ================================================================================ {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 13 01:04:16.494173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ================================================================================ {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 13 01:04:16.494312 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] admin_token = **** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.494421 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] admin_token = **** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.494560 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] config_dir = [] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.494652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] config_dir = [] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.494732 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.494845 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.494954 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] config_source = [] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.495053 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] config_source = [] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.495135 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] debug = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.495244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] debug = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.495359 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.496529 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.496654 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.496734 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.496800 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.496858 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.496914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.496969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.497029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.497096 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.497159 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] list_limit = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.497236 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] list_limit = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.497236 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] log_config_append = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.497334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] log_config_append = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.497388 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.497453 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.497516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] log_dir = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.497572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] log_dir = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.497626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] log_file = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.497685 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] log_file = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.497741 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.497805 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.497861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.497920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.497981 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.498036 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.498095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.498161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.498338 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.498423 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.498487 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.498550 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.498704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.498776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.498965 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.499040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.499174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.499260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.499403 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.499481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.499622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.499698 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.499854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.499936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.500101 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.500185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.500346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.500444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.500618 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.500715 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.501106 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.15/identity {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.501192 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.15/identity {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.501306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] publish_errors = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.501392 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] publish_errors = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.501520 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.501596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.501749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.501835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.501985 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.502064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.502172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.502267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.502401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.502492 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.502627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] standard_threads = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.502704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] standard_threads = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.502876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.502966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.503125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.503202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.503307 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.503426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.503538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] use_journal = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.503648 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] use_journal = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.503805 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] use_json = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.503919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] use_json = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.504029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] use_stderr = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.504139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] use_stderr = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.504256 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] use_syslog = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.504366 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] use_syslog = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.504485 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.504603 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 01:04:16.504720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.504831 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.504945 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.505055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.505173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.505290 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.505401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.505512 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.505622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.505737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.505858 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.505970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.506079 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.506194 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.506305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.506414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.506532 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.506644 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.506760 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.506890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.506890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.507100 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.507100 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.507310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.507473 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.507623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.507734 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.507850 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.507970 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.508082 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.508192 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.508302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.508454 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.508563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.508683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.508802 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.508913 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.509107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.509234 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.509345 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.509533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.509648 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.509767 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.509877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.509994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.510123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.510245 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.510357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.510467 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.510577 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.510687 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.510810 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.510969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.511081 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.511202 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.511315 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.511431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.511551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.511669 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.511788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.511900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.512009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.512126 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.512245 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.512357 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.512467 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.512610 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.512730 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.512849 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.512968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.513080 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.513222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.513341 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.513452 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.513562 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.513683 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.513796 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.513907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.514016 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.514126 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.514241 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.514361 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.514472 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.514581 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.514697 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.514809 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.514935 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.515176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.515294 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.515405 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.515515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.515631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.515743 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.515955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.516072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.516183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.516294 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.516404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.516513 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.516632 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.516745 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.516855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.516980 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.517197 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.517304 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.517412 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.517507 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.517599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.517691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.517783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.517882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.517984 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.518099 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.518209 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.518352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.518508 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.518631 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.518743 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.518845 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.518988 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.519101 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.519214 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.519333 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.519446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.519557 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.519676 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.519797 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.519916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.520027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.520139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.520327 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.520450 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.520563 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.520675 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.520786 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.520903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.521027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.521132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.521244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.521354 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.521471 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.521584 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.521702 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.521814 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.521925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.522036 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.522147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.522265 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.522390 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.522494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.522604 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.522721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.522832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.522964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.523088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.523202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.523321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.523433 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.523544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.523655 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.523774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.523893 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.524011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.524123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.524234 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.524345 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.524465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.524578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.524689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.525317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.525452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.525564 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.525685 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.525805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.525919 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.526030 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.526150 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.526265 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.526265 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.526486 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.526598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.526709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.526820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.526979 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.527172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.527288 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.527400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.527512 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.527633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.527753 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.527867 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.527990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.528104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.528217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.528329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.528441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.528592 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.528720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.528835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.528953 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.529073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.529185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.529298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.529417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.529531 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.529643 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.529755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.529866 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.529984 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.530105 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.530219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.530330 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.530442 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.530584 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.530708 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] auth.external = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.530835 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] auth.external = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.530969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.531094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.531209 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.531323 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.531437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.531556 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.531670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] auth.password = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.531792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] auth.password = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.531906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] auth.token = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.532030 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] auth.token = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.532145 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.532258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.532370 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.532494 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.532608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.532720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.532834 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.532952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.533065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.533186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.533300 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.533418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.533532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.533644 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.533757 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] credential.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.533879 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] credential.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.533993 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.534105 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.534217 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.534329 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.534441 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.534562 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.534676 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.534788 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.534950 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.535065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.535188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.535315 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.535433 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.535546 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.535665 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.535778 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.535890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.536011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.536131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.536245 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.536357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.536470 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.536582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.536702 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.536816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.536928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.537042 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.537154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.537267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.537388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.537502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] federation.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.537614 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] federation.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.537727 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.537838 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.537951 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.538072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.538186 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.538299 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.538411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.538523 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.538666 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.538790 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.538917 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.539031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.539145 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.539257 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.539382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.539497 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.539609 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.539721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.539842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.539956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.540076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.540191 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.540304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.541053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.541178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.541291 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.541481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.541598 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.541711 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.541824 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.541936 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.542049 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.542173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.542289 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.542401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.542513 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.542633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.542747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.542897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.543015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.543128 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.543241 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.543354 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.543466 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.543591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.543706 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.543819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.543931 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.544044 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.544156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.544276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.544391 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.544510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.544623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.544736 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.544849 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.545294 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.545415 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.545534 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.545647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.545761 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.545873 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.545995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.546109 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.546223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.546335 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.546449 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.546561 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.546683 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.546797 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.546926 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.547040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.547164 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.547279 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.547401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.547516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.547629 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.547749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.547863 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.547976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.548097 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.548211 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.548324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.548443 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.548559 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.548718 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.548845 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.548959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.549072 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.549184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.549297 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.549409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.549529 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.549644 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.549756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.549868 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.549981 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.550101 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.550215 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.550327 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.550440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.550552 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.550665 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.550785 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.550938 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.551053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.551167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.551290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.551404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.551525 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.551648 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.551762 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.551889 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.552060 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.552177 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.552299 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.552414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.552527 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.552640 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.552760 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.552874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.552995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.553109 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.553222 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.553334 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.553446 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.553559 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.553685 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.553801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.553914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.554027 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.554139 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.554251 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.554372 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.554487 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.554599 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.554712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.554825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.554954 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.555079 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.555195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.555307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.555421 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.555538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.555658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.555781 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.555896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.556008 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.556121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.556233 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.556346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.556467 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.556581 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.556693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.556806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.556919 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.557031 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.557153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.557267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.557379 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.557492 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.557604 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.557717 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.557837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.557953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.558065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.558178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.558290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.558402 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.558523 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.558637 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.558785 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.558949 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.559065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.559179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.559303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.559418 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.559531 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.559644 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.559755 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.559868 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.559989 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.560104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.560216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.560329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.560441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.560554 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.560675 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.560789 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.560901 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.561013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.561127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.561240 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.561619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.561738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.561850 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.561963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.562076 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.562189 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] resource.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.562310 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] resource.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.562425 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.562537 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.562650 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.562762 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.563110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.563242 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.563358 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.563471 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.563584 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.563696 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.563809 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.563930 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.564044 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.564156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.564270 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.564382 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.564494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.564683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.564799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] role.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.564912 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] role.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.565025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] role.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.565137 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] role.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.565257 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.565371 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.565484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.565600 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.565713 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.565825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.565946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.566060 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.566173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.566286 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.566399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.566512 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.566632 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.566746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.566877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.566993 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.567106 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.567218 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.567342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.567458 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.567570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.567682 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.567794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.567907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.568027 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.568141 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.568253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.568366 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.568480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.568592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.568713 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.568864 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.568980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.569092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.569205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.569317 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.569438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.569554 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.569666 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.569785 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.569899 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.570019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.570133 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.570246 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.570359 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.570471 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.570585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.570705 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.570819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.570961 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.571076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.571188 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.571301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.571425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.571540 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.572169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.572284 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.572396 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.572509 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.572631 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.572746 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.572859 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.572971 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.573084 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.573197 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.573318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.573432 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] token.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.573546 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] token.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.573659 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.573772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.573885 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.574006 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.574120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.574233 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.574346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.574459 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.574573 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.574693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.574808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.574945 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.575059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.575171 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.575283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.575408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.575523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.575635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.575748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.575861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.575973 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.576096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.576217 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.576330 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.576442 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.576554 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.576666 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.576786 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.576901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.577022 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.577149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.577271 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.577385 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.577497 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.577610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.connection = **** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.577722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.connection = **** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.577843 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.577958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.578070 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.578182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.578295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.578407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.578528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.578641 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.578754 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.578936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.579055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.579168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.579298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.579413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.579526 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.579638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.579751 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.579863 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.579985 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.580089 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.580200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.580312 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.580424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.580536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.580656 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.580770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.580883 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.580996 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.581109 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.581221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.581341 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.581454 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.581567 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.581679 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.581792 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.581792 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81405) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 13 01:04:16.582012 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 01:04:16.582012 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81404) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 13 01:04:16.612324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.612719 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.612995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.616287 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.616521 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.616746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.621451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.625185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.650511 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.650738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.652188 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.652407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.703970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.704362 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.753350 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.753581 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.754404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.754642 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.907995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.908202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.908328 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.908415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.908561 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.908642 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.908783 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.908865 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.909004 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.909084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.909220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.909432 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.933054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.933283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.933463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.933552 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.933700 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.933799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.933914 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.933993 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.934129 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.934209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.934343 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.934423 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.934557 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.934769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.945776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.946046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.958536 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.958721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.959039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.959161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.959292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.959411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.959520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.959740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.959829 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.959956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.960140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.960221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.960370 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81404) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.960513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81405) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 01:04:16.978573 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.978774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 13 01:04:16.979146 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.979348 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 13 01:04:16.979509 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.979621 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.979701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 13 01:04:16.979882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.980027 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.980139 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.980218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 13 01:04:16.980393 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.980530 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.980747 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.981007 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.981270 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.989933 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.990212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.990484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.990769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.990867 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.991123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.991398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.991680 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:16.997766 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:16.998648 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.000702 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.000989 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.001274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.001557 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.001637 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.001841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.001920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.002101 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.002189 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.002373 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.002454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.002646 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.002725 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.002940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.003060 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.003212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.003289 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.003480 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.003572 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.003752 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.004026 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.004296 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.026648 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.027568 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.028150 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.028437 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.029048 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.029342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.033112 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.033400 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.033683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.034095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.034381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.034662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.040066 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.040375 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.041711 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.041995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.046572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.046888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.047178 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.047461 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.047690 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.047772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.047987 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.048077 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.048258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.048337 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.048527 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.048607 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.048803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.049078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.049349 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.049616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.064196 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.064473 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.065302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.065581 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.068303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.069388 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.071493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.071817 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.072623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.072922 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.075973 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.077080 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.079195 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.080173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.082417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.082721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.083028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.083316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.083398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.083611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.083692 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.083894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.083982 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.084198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.084479 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.084760 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.100166 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.100457 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.100727 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.100885 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.100990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.101172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.101442 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.101713 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.108419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.109036 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.109958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.110550 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.111516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.111795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.112088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.112366 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.114496 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.114783 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.115096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.115178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.115382 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.115463 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.115701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.115792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.115986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.116380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.126941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.127229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.127513 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.127621 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.127798 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.127904 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.128182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.128264 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.128457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.128537 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.128740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.128819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.129101 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.129389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.144147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.144832 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.147077 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.147757 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.150520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.151225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.152033 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.152680 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.153488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.153772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.154128 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.154208 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.154410 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.154802 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.160556 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.161255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.162013 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.162718 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.164909 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.165307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.166158 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.166249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.166438 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.166545 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.166833 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.167135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.174266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.174555 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.174847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.174938 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.175222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.175504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.181771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81404) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.182057 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.182339 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.182431 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81405) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 01:04:17.182619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.182699 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.182915 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.182997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.183193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.183286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.183470 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.183549 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.183742 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.183822 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.184024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.184115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.184293 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.184372 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.184559 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.184639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.184827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.184905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.185094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.185217 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.185351 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.185430 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81404) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.185625 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.185907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81405) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 01:04:17.220025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5571e23a69c0 pid: 81404 (default app) Sep 13 01:04:17.220382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5571e23a69c0 pid: 81405 (default app) Sep 13 01:04:17.223216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b66e206-dd6e-41c6-a3ad-937065bf9e0a None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:17.223503 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b66e206-dd6e-41c6-a3ad-937065bf9e0a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:17.223777 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b66e206-dd6e-41c6-a3ad-937065bf9e0a None None] PATH_INFO: `/v3` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:17.227893 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1/1] 10.222.0.15 () {50 vars in 791 bytes} [Tue Sep 13 01:04:17 2022] GET /identity/v3/ => generated 255 bytes in 7 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:23.965900 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc4a8e3-c85b-4007-98e8-9a1324095ed3 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:23.966412 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc4a8e3-c85b-4007-98e8-9a1324095ed3 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:23.966412 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc4a8e3-c85b-4007-98e8-9a1324095ed3 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:23.970679 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1/2] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:23 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:04:23.976021 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b01715c-c80d-4803-9d72-72dd906b40d4 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:23.976606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b01715c-c80d-4803-9d72-72dd906b40d4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:23.976606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b01715c-c80d-4803-9d72-72dd906b40d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:24.045508 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: INFO dbcounter [None req-5b01715c-c80d-4803-9d72-72dd906b40d4 None None] Registered counter for database keystone Sep 13 01:04:24.064921 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG oslo_db.sqlalchemy.engines [None req-5b01715c-c80d-4803-9d72-72dd906b40d4 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81404) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 13 01:04:24.071100 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writer thread running {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 13 01:04:24.249560 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG passlib.handlers.bcrypt [None req-5b01715c-c80d-4803-9d72-72dd906b40d4 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81404) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 13 01:04:24.249941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG passlib.handlers.bcrypt [None req-5b01715c-c80d-4803-9d72-72dd906b40d4 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81404) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 13 01:04:24.308378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-5b01715c-c80d-4803-9d72-72dd906b40d4 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:24.309996 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5b01715c-c80d-4803-9d72-72dd906b40d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:24.439023 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 2/3] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:23 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 464 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Sep 13 01:04:24.915853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc4a8e3-c85b-4007-98e8-9a1324095ed3 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:24.983487 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: INFO dbcounter [None req-5fc4a8e3-c85b-4007-98e8-9a1324095ed3 None None] Registered counter for database keystone Sep 13 01:04:25.004718 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG oslo_db.sqlalchemy.engines [None req-5fc4a8e3-c85b-4007-98e8-9a1324095ed3 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81405) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 13 01:04:25.011572 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writer thread running {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 13 01:04:25.129201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc4a8e3-c85b-4007-98e8-9a1324095ed3 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:25.133464 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28cf29a-6c51-4091-87b0-996d0a7d1cdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:25.134100 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28cf29a-6c51-4091-87b0-996d0a7d1cdc admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:25.134381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28cf29a-6c51-4091-87b0-996d0a7d1cdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:25.134652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28cf29a-6c51-4091-87b0-996d0a7d1cdc admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:25.142755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28cf29a-6c51-4091-87b0-996d0a7d1cdc admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:25.194502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: WARNING keystone.common.rbac_enforcer.enforcer [None req-d28cf29a-6c51-4091-87b0-996d0a7d1cdc admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 13 01:04:25.194792 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28cf29a-6c51-4091-87b0-996d0a7d1cdc admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:25.203987 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 2/4] 10.222.0.15 () {58 vars in 1324 bytes} [Tue Sep 13 01:04:24 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 01:04:25.209925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b01715c-c80d-4803-9d72-72dd906b40d4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:25.226901 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b01715c-c80d-4803-9d72-72dd906b40d4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:25.231839 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4289b4-63cd-4756-b31f-f96aac1b5ee1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:25.232937 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4289b4-63cd-4756-b31f-f96aac1b5ee1 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:25.233216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4289b4-63cd-4756-b31f-f96aac1b5ee1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:25.233483 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4289b4-63cd-4756-b31f-f96aac1b5ee1 admin admin] PATH_INFO: `/v3/projects/ce37ba2e4948471599c41255ecf54622` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:25.235072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4289b4-63cd-4756-b31f-f96aac1b5ee1 admin admin] RBAC: Authorizing `identity:get_project(project_id=ce37ba2e4948471599c41255ecf54622)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:25.287080 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: WARNING keystone.common.rbac_enforcer.enforcer [None req-1b4289b4-63cd-4756-b31f-f96aac1b5ee1 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 13 01:04:25.287418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4289b4-63cd-4756-b31f-f96aac1b5ee1 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:25.292391 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 3/5] 10.222.0.15 () {56 vars in 1207 bytes} [Tue Sep 13 01:04:25 2022] GET /identity/v3/projects/ce37ba2e4948471599c41255ecf54622 => generated 344 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:27.389397 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e40c95c-db44-45ec-bffb-24bedf13f9a1 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:27.389969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e40c95c-db44-45ec-bffb-24bedf13f9a1 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:27.389969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e40c95c-db44-45ec-bffb-24bedf13f9a1 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:27.394136 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 3/6] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:27 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:27.399621 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7010df-4de8-41b5-b631-a07894e35186 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:27.400646 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7010df-4de8-41b5-b631-a07894e35186 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:27.400738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7010df-4de8-41b5-b631-a07894e35186 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:27.420804 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-2f7010df-4de8-41b5-b631-a07894e35186 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:27.422308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2f7010df-4de8-41b5-b631-a07894e35186 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:27.430111 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 4/7] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:27 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 13 01:04:27.706689 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e40c95c-db44-45ec-bffb-24bedf13f9a1 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:27.719097 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e40c95c-db44-45ec-bffb-24bedf13f9a1 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:27.723237 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a63158-4d32-44d6-9ebf-4eab2c0ad3bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:27.723831 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a63158-4d32-44d6-9ebf-4eab2c0ad3bc admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:27.724107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a63158-4d32-44d6-9ebf-4eab2c0ad3bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:27.724377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a63158-4d32-44d6-9ebf-4eab2c0ad3bc admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:27.732440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a63158-4d32-44d6-9ebf-4eab2c0ad3bc admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:27.734174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a63158-4d32-44d6-9ebf-4eab2c0ad3bc admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:27.742722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 4/8] 10.222.0.15 () {58 vars in 1324 bytes} [Tue Sep 13 01:04:27 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 01:04:27.747911 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7010df-4de8-41b5-b631-a07894e35186 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:27.756781 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7010df-4de8-41b5-b631-a07894e35186 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:27.761031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0d969a-2aa0-42d5-b99b-61fb19c6811f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:27.761644 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0d969a-2aa0-42d5-b99b-61fb19c6811f admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:27.761921 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0d969a-2aa0-42d5-b99b-61fb19c6811f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:27.762190 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0d969a-2aa0-42d5-b99b-61fb19c6811f admin admin] PATH_INFO: `/v3/users/22ddb5151c0f4696a4f99a9acc8804ea` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:27.763502 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0d969a-2aa0-42d5-b99b-61fb19c6811f admin admin] RBAC: Authorizing `identity:get_user(user_id=22ddb5151c0f4696a4f99a9acc8804ea)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:27.765300 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0d969a-2aa0-42d5-b99b-61fb19c6811f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:27.766801 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 5/9] 10.222.0.15 () {56 vars in 1198 bytes} [Tue Sep 13 01:04:27 2022] GET /identity/v3/users/22ddb5151c0f4696a4f99a9acc8804ea => generated 247 bytes in 19 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:29.963668 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4612cca5-26c9-4ddf-98bf-b5be5027e8ad None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:29.964197 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4612cca5-26c9-4ddf-98bf-b5be5027e8ad None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:29.964197 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4612cca5-26c9-4ddf-98bf-b5be5027e8ad None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:29.965517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 5/10] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:29.970822 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1818c07a-b518-4054-b761-b6c1df1e5ea4 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:29.971252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1818c07a-b518-4054-b761-b6c1df1e5ea4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:29.971405 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1818c07a-b518-4054-b761-b6c1df1e5ea4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:29.983381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c133a9bc-0f09-445a-83b6-5746cb2fe672 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:29.983667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c133a9bc-0f09-445a-83b6-5746cb2fe672 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:29.983946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c133a9bc-0f09-445a-83b6-5746cb2fe672 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:29.985553 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 6/11] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:04:29.989826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-1818c07a-b518-4054-b761-b6c1df1e5ea4 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:29.990257 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45653358-62b4-4252-b1b8-299294d51081 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:29.990542 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45653358-62b4-4252-b1b8-299294d51081 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:29.990820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45653358-62b4-4252-b1b8-299294d51081 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:29.991023 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1818c07a-b518-4054-b761-b6c1df1e5ea4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:30.032900 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG passlib.handlers.bcrypt [None req-45653358-62b4-4252-b1b8-299294d51081 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81405) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 13 01:04:30.033290 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG passlib.handlers.bcrypt [None req-45653358-62b4-4252-b1b8-299294d51081 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81405) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 13 01:04:30.070997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 6/12] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:30.082826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-45653358-62b4-4252-b1b8-299294d51081 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:30.084444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-45653358-62b4-4252-b1b8-299294d51081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:30.184358 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 7/13] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 01:04:30.344117 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1818c07a-b518-4054-b761-b6c1df1e5ea4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:30.369922 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:30.448784 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45653358-62b4-4252-b1b8-299294d51081 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:30.453716 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1818c07a-b518-4054-b761-b6c1df1e5ea4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:30.487329 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f419400c-bc29-4c9f-83ec-a3bb992f9dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:30.487967 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f419400c-bc29-4c9f-83ec-a3bb992f9dfd None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:30.488253 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f419400c-bc29-4c9f-83ec-a3bb992f9dfd None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:30.488533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f419400c-bc29-4c9f-83ec-a3bb992f9dfd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:30.499501 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f419400c-bc29-4c9f-83ec-a3bb992f9dfd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:04:30.508250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-f419400c-bc29-4c9f-83ec-a3bb992f9dfd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:04:30.508250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:04:30.508250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:04:30.508250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:04:30.508250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:04:30.508250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:04:30.508250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:04:30.508250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:04:30.508250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:04:30.508250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:04:30.508250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:04:30.508250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:04:30.508250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:04:30.508250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:04:30.508250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:04:30.508250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:04:30.509290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:04:30.509290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:04:30.509290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:04:30.509290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:04:30.509290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:04:30.509290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:04:30.509290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:04:30.509290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:04:30.509290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:04:30.509290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:04:30.509290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:04:30.509290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:04:30.509290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:04:30.509290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:04:30.509290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:04:30.509290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:04:30.510179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:04:30.510179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:04:30.510179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:04:30.510179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:04:30.510179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:04:30.510179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:04:30.510179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:04:30.510179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:04:30.510179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:04:30.510179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:04:30.510179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:04:30.510179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:04:30.510179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 7/14] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:04:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:04:30.515241 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f419400c-bc29-4c9f-83ec-a3bb992f9dfd None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:30.551747 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45653358-62b4-4252-b1b8-299294d51081 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:30.554027 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:30.591188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d264b8d1-3031-424a-8607-3ff3a36fa488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:30.592048 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d264b8d1-3031-424a-8607-3ff3a36fa488 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:30.592708 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d264b8d1-3031-424a-8607-3ff3a36fa488 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:30.592984 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d264b8d1-3031-424a-8607-3ff3a36fa488 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:30.597004 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:30.616168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f419400c-bc29-4c9f-83ec-a3bb992f9dfd None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:30.649882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6765cd68-36a4-4368-b5e5-7fb1d467c072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:30.650482 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6765cd68-36a4-4368-b5e5-7fb1d467c072 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:30.650758 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6765cd68-36a4-4368-b5e5-7fb1d467c072 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:30.651061 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6765cd68-36a4-4368-b5e5-7fb1d467c072 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:30.651713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6765cd68-36a4-4368-b5e5-7fb1d467c072 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:30.652012 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6765cd68-36a4-4368-b5e5-7fb1d467c072 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:30.653382 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6765cd68-36a4-4368-b5e5-7fb1d467c072 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:30.659989 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d264b8d1-3031-424a-8607-3ff3a36fa488 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:30.661483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d264b8d1-3031-424a-8607-3ff3a36fa488 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:30.664725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 8/15] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:04:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:30.670388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6765cd68-36a4-4368-b5e5-7fb1d467c072 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:30.758599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 8/16] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:04:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 13 01:04:30.764297 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d264b8d1-3031-424a-8607-3ff3a36fa488 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:30.766912 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:30.767023 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6765cd68-36a4-4368-b5e5-7fb1d467c072 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:30.799995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a433b7-bb40-4224-b95b-ed5696062d00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:30.800621 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a433b7-bb40-4224-b95b-ed5696062d00 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:30.800898 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a433b7-bb40-4224-b95b-ed5696062d00 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:30.801167 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a433b7-bb40-4224-b95b-ed5696062d00 None admin] PATH_INFO: `/v3/users/22ddb5151c0f4696a4f99a9acc8804ea` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:30.802295 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a433b7-bb40-4224-b95b-ed5696062d00 None admin] RBAC: Authorizing `identity:get_user(user_id=22ddb5151c0f4696a4f99a9acc8804ea)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:30.803767 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a433b7-bb40-4224-b95b-ed5696062d00 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:30.805307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 9/17] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:04:30 2022] GET /identity/v3/users/22ddb5151c0f4696a4f99a9acc8804ea => generated 247 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:30.811128 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a433b7-bb40-4224-b95b-ed5696062d00 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:30.826648 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:30.862786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d264b8d1-3031-424a-8607-3ff3a36fa488 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:30.895954 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf1055e-655c-4c6c-b340-1ab8d681bc39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:30.896802 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf1055e-655c-4c6c-b340-1ab8d681bc39 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:30.897084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf1055e-655c-4c6c-b340-1ab8d681bc39 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:30.897360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf1055e-655c-4c6c-b340-1ab8d681bc39 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:30.908820 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a433b7-bb40-4224-b95b-ed5696062d00 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:30.915837 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf1055e-655c-4c6c-b340-1ab8d681bc39 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:30.917283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf1055e-655c-4c6c-b340-1ab8d681bc39 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:30.935435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-1cf1055e-655c-4c6c-b340-1ab8d681bc39 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:04:30.935435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:04:30.935435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:04:30.935435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:04:30.935435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:04:30.935435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:04:30.935435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:04:30.935435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:04:30.935435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:04:30.935435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:04:30.935435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:04:30.935435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:04:30.935435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:04:30.935435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:04:30.935435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:04:30.935435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:04:30.936303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:04:30.936303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:04:30.936303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:04:30.936303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:04:30.936303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:04:30.936303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:04:30.936303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:04:30.936303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:04:30.936303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:04:30.936303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:04:30.936303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:04:30.936303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:04:30.936303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:04:30.936303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:04:30.936303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:04:30.936303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:04:30.936303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:04:30.937768 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:04:30.937768 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:04:30.937768 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:04:30.937768 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 9/18] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:04:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:04:30.942446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf1055e-655c-4c6c-b340-1ab8d681bc39 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:30.942735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d770dce-efd7-4510-876c-39df5af60091 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:30.943388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d770dce-efd7-4510-876c-39df5af60091 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:30.943663 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d770dce-efd7-4510-876c-39df5af60091 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:30.943948 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d770dce-efd7-4510-876c-39df5af60091 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:30.945044 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d770dce-efd7-4510-876c-39df5af60091 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:30.946469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d770dce-efd7-4510-876c-39df5af60091 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:30.948554 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 10/19] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:04:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:30.954100 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d770dce-efd7-4510-876c-39df5af60091 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:30.999515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:31.032718 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:31.045238 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf1055e-655c-4c6c-b340-1ab8d681bc39 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:31.052673 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d770dce-efd7-4510-876c-39df5af60091 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:31.079711 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb950745-44f4-46e6-9138-6a1fa7651fb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:31.080314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb950745-44f4-46e6-9138-6a1fa7651fb4 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:31.080592 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb950745-44f4-46e6-9138-6a1fa7651fb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:31.080870 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb950745-44f4-46e6-9138-6a1fa7651fb4 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:31.081498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb950745-44f4-46e6-9138-6a1fa7651fb4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:31.081800 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb950745-44f4-46e6-9138-6a1fa7651fb4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:31.083194 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb950745-44f4-46e6-9138-6a1fa7651fb4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:31.086326 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f36dbe3-9aa0-4188-a112-1f6a2657123f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:31.087400 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f36dbe3-9aa0-4188-a112-1f6a2657123f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:31.087687 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f36dbe3-9aa0-4188-a112-1f6a2657123f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:31.087962 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f36dbe3-9aa0-4188-a112-1f6a2657123f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:31.088680 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f36dbe3-9aa0-4188-a112-1f6a2657123f None admin] RBAC: Adding query filter params (role.id=66a36c858aab40688080eed6ceace3a0, scope.domain.id=default, user.id=22ddb5151c0f4696a4f99a9acc8804ea) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:31.088986 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f36dbe3-9aa0-4188-a112-1f6a2657123f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:31.090402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f36dbe3-9aa0-4188-a112-1f6a2657123f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:31.098322 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 11/20] 10.222.0.15 () {56 vars in 1322 bytes} [Tue Sep 13 01:04:30 2022] GET /identity/v3/role_assignments?user.id=22ddb5151c0f4696a4f99a9acc8804ea&scope.domain.id=default&role.id=66a36c858aab40688080eed6ceace3a0 => generated 233 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:31.100063 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 10/21] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:04:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:33.503901 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354ca6fc-2b00-4af5-ba41-4ef557b69021 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:33.504882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354ca6fc-2b00-4af5-ba41-4ef557b69021 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:33.505402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354ca6fc-2b00-4af5-ba41-4ef557b69021 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:33.507236 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 12/22] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:33.513119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa72c41-e15b-49e5-b509-e7ea39440237 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:33.513911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa72c41-e15b-49e5-b509-e7ea39440237 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:33.514473 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa72c41-e15b-49e5-b509-e7ea39440237 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:33.537504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9aa72c41-e15b-49e5-b509-e7ea39440237 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:33.539037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9aa72c41-e15b-49e5-b509-e7ea39440237 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:33.619237 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 11/23] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 01:04:33.942882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45178ceb-2d96-454f-8de0-0f3fecf9130b None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:33.943599 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45178ceb-2d96-454f-8de0-0f3fecf9130b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:33.944135 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45178ceb-2d96-454f-8de0-0f3fecf9130b None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:33.945671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 13/24] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:33.951715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de7ad0f-317b-4102-af18-2b1c3ca9106e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:33.952238 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de7ad0f-317b-4102-af18-2b1c3ca9106e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:33.952711 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de7ad0f-317b-4102-af18-2b1c3ca9106e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:33.977513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8de7ad0f-317b-4102-af18-2b1c3ca9106e None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:33.979662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8de7ad0f-317b-4102-af18-2b1c3ca9106e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:34.016529 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45178ceb-2d96-454f-8de0-0f3fecf9130b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:34.051525 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:34.098747 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 12/25] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:34.100141 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03484856-9ec4-4bea-a42a-0b5962901890 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:34.103667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03484856-9ec4-4bea-a42a-0b5962901890 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:34.103962 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03484856-9ec4-4bea-a42a-0b5962901890 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:34.105632 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 13/26] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:34 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:34.111059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cfef1b-259c-4b9f-8b6a-b9c847577cb8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:34.111376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cfef1b-259c-4b9f-8b6a-b9c847577cb8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:34.111882 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cfef1b-259c-4b9f-8b6a-b9c847577cb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:34.142447 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-63cfef1b-259c-4b9f-8b6a-b9c847577cb8 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:34.144823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-63cfef1b-259c-4b9f-8b6a-b9c847577cb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:34.237814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45178ceb-2d96-454f-8de0-0f3fecf9130b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:34.258230 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 14/27] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:34.287448 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615f2789-e940-4c26-8e1e-d8d317da8616 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:34.288311 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615f2789-e940-4c26-8e1e-d8d317da8616 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:34.288740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615f2789-e940-4c26-8e1e-d8d317da8616 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:34.289156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615f2789-e940-4c26-8e1e-d8d317da8616 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:34.300548 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615f2789-e940-4c26-8e1e-d8d317da8616 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:04:34.306619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-615f2789-e940-4c26-8e1e-d8d317da8616 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:04:34.306619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:04:34.306619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:04:34.306619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:04:34.306619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:04:34.306619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:04:34.306619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:04:34.306619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:04:34.306619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:04:34.306619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:04:34.306619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:04:34.306619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:04:34.306619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:04:34.306619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:04:34.306619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:04:34.306619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:04:34.307853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:04:34.307853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:04:34.307853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:04:34.307853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:04:34.307853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:04:34.307853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:04:34.307853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:04:34.307853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:04:34.307853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:04:34.307853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:04:34.307853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:04:34.307853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:04:34.307853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:04:34.307853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:04:34.307853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:04:34.307853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:04:34.308828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:04:34.308828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:04:34.308828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:04:34.308828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:04:34.308828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:04:34.308828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:04:34.308828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:04:34.308828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:04:34.308828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:04:34.308828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:04:34.308828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:04:34.308828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:04:34.310715 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 14/28] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:04:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:04:34.317450 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cfef1b-259c-4b9f-8b6a-b9c847577cb8 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:34.457781 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cfef1b-259c-4b9f-8b6a-b9c847577cb8 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:34.467740 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:34.500415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096f2c67-f29d-45cf-8089-af463bcdd3d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:34.501287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096f2c67-f29d-45cf-8089-af463bcdd3d8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:34.501763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096f2c67-f29d-45cf-8089-af463bcdd3d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:34.502211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096f2c67-f29d-45cf-8089-af463bcdd3d8 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:34.503083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096f2c67-f29d-45cf-8089-af463bcdd3d8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:34.503562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096f2c67-f29d-45cf-8089-af463bcdd3d8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:34.505112 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096f2c67-f29d-45cf-8089-af463bcdd3d8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:34.522400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 15/29] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:04:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:34.528796 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615f2789-e940-4c26-8e1e-d8d317da8616 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:34.533166 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:34.635316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615f2789-e940-4c26-8e1e-d8d317da8616 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:34.668280 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096f2c67-f29d-45cf-8089-af463bcdd3d8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:34.684383 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0044e8d0-ead1-4f8f-b37a-8c01c4f3272b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:34.685485 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0044e8d0-ead1-4f8f-b37a-8c01c4f3272b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:34.687818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0044e8d0-ead1-4f8f-b37a-8c01c4f3272b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:34.688287 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0044e8d0-ead1-4f8f-b37a-8c01c4f3272b None admin] PATH_INFO: `/v3/users/22ddb5151c0f4696a4f99a9acc8804ea` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:34.689661 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0044e8d0-ead1-4f8f-b37a-8c01c4f3272b None admin] RBAC: Authorizing `identity:get_user(user_id=22ddb5151c0f4696a4f99a9acc8804ea)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:34.692188 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0044e8d0-ead1-4f8f-b37a-8c01c4f3272b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:34.694776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 15/30] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:04:34 2022] GET /identity/v3/users/22ddb5151c0f4696a4f99a9acc8804ea => generated 247 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:34.701800 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0044e8d0-ead1-4f8f-b37a-8c01c4f3272b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:34.835167 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:34.861709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096f2c67-f29d-45cf-8089-af463bcdd3d8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:34.868315 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0044e8d0-ead1-4f8f-b37a-8c01c4f3272b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:34.916529 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24ec71a-03a3-45fd-a8eb-7c90f5e345f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:34.917170 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24ec71a-03a3-45fd-a8eb-7c90f5e345f9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:34.917493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24ec71a-03a3-45fd-a8eb-7c90f5e345f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:34.917719 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24ec71a-03a3-45fd-a8eb-7c90f5e345f9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:34.918864 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24ec71a-03a3-45fd-a8eb-7c90f5e345f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:34.920422 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24ec71a-03a3-45fd-a8eb-7c90f5e345f9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:34.922075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 16/31] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:04:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:34.922381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb6502a-2673-4102-b07a-ca52e01642f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:34.923271 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb6502a-2673-4102-b07a-ca52e01642f8 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:34.923759 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb6502a-2673-4102-b07a-ca52e01642f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:34.924254 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb6502a-2673-4102-b07a-ca52e01642f8 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:34.925166 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb6502a-2673-4102-b07a-ca52e01642f8 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:34.926797 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb6502a-2673-4102-b07a-ca52e01642f8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:34.927776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24ec71a-03a3-45fd-a8eb-7c90f5e345f9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:35.049751 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24ec71a-03a3-45fd-a8eb-7c90f5e345f9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:35.080909 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 16/32] 10.222.0.15 () {60 vars in 1131 bytes} [Tue Sep 13 01:04:34 2022] POST /identity/v3/roles => generated 219 bytes in 413 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:04:35.085732 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f1094b-941a-472a-9415-a3f6bac4274f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:35.086314 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f1094b-941a-472a-9415-a3f6bac4274f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:35.086597 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f1094b-941a-472a-9415-a3f6bac4274f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:35.086903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f1094b-941a-472a-9415-a3f6bac4274f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:35.089500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4fc23c-97cb-4fcf-8699-504c9cbe7800 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:35.090126 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4fc23c-97cb-4fcf-8699-504c9cbe7800 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:35.091260 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4fc23c-97cb-4fcf-8699-504c9cbe7800 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:35.092963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 17/33] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:35 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:35.093922 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4fc23c-97cb-4fcf-8699-504c9cbe7800 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:35.099086 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:35.102271 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f1094b-941a-472a-9415-a3f6bac4274f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:35.103943 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f1094b-941a-472a-9415-a3f6bac4274f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:35.122515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-21f1094b-941a-472a-9415-a3f6bac4274f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:04:35.122515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:04:35.122515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:04:35.122515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:04:35.122515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:04:35.122515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:04:35.122515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:04:35.122515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:04:35.122515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:04:35.122515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:04:35.122515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:04:35.122515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:04:35.122515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:04:35.122515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:04:35.122515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:04:35.122515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:04:35.123639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:04:35.123639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:04:35.123639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:04:35.123639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:04:35.123639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:04:35.123639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:04:35.123639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:04:35.123639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:04:35.123639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:04:35.123639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:04:35.123639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:04:35.123639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:04:35.123639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:04:35.123639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:04:35.123639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:04:35.123639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:04:35.123639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:04:35.124490 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:04:35.124490 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:04:35.124490 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:04:35.125252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 17/34] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:04:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:04:35.126735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e16f0e1-6c51-449e-957f-9ad52d4283fe None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:35.127124 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e16f0e1-6c51-449e-957f-9ad52d4283fe None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:35.127476 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e16f0e1-6c51-449e-957f-9ad52d4283fe None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:35.133096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 18/35] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:35.133096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b67f3a5-7e75-4989-b598-93e0d57fae57 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:35.133096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b67f3a5-7e75-4989-b598-93e0d57fae57 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:35.133096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b67f3a5-7e75-4989-b598-93e0d57fae57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:35.145584 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:35.151245 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8b67f3a5-7e75-4989-b598-93e0d57fae57 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:35.152657 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8b67f3a5-7e75-4989-b598-93e0d57fae57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:35.248511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 19/36] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:35.250225 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b67f3a5-7e75-4989-b598-93e0d57fae57 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:35.253847 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4fc23c-97cb-4fcf-8699-504c9cbe7800 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:35.295218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f475c5-941d-4a21-b9be-bc805dbe26fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:35.296015 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f475c5-941d-4a21-b9be-bc805dbe26fe None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:35.296304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f475c5-941d-4a21-b9be-bc805dbe26fe None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:35.296584 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f475c5-941d-4a21-b9be-bc805dbe26fe None admin] PATH_INFO: `/v3/domains/default/users/22ddb5151c0f4696a4f99a9acc8804ea/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:35.300512 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f475c5-941d-4a21-b9be-bc805dbe26fe None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=22ddb5151c0f4696a4f99a9acc8804ea, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:35.302109 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f475c5-941d-4a21-b9be-bc805dbe26fe None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:35.326826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 18/37] 10.222.0.15 () {58 vars in 1361 bytes} [Tue Sep 13 01:04:35 2022] PUT /identity/v3/domains/default/users/22ddb5151c0f4696a4f99a9acc8804ea/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:04:35.328928 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1780110-dda7-4202-a3bc-e5569945e899 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:35.328928 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1780110-dda7-4202-a3bc-e5569945e899 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:35.328928 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1780110-dda7-4202-a3bc-e5569945e899 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:35.349808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a1780110-dda7-4202-a3bc-e5569945e899 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:35.351303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a1780110-dda7-4202-a3bc-e5569945e899 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:35.392781 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b67f3a5-7e75-4989-b598-93e0d57fae57 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:35.424642 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:35.441426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17eefee-8876-4201-b382-a39cadac6ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:35.442502 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17eefee-8876-4201-b382-a39cadac6ce7 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:35.443055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17eefee-8876-4201-b382-a39cadac6ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:35.443656 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17eefee-8876-4201-b382-a39cadac6ce7 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:35.444630 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17eefee-8876-4201-b382-a39cadac6ce7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:35.445179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17eefee-8876-4201-b382-a39cadac6ce7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:35.447077 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17eefee-8876-4201-b382-a39cadac6ce7 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:35.463763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 19/38] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:35.469909 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 20/39] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:04:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:35.476870 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1780110-dda7-4202-a3bc-e5569945e899 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:35.503903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:35.545922 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8d7ef4-81d7-4cda-8e84-5cf46e417f84 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:35.546930 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8d7ef4-81d7-4cda-8e84-5cf46e417f84 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:35.547474 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8d7ef4-81d7-4cda-8e84-5cf46e417f84 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:35.549304 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 21/40] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:35.557793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d035905-98c8-43e5-81c5-665aca7b2787 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:35.558444 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d035905-98c8-43e5-81c5-665aca7b2787 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:35.558971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d035905-98c8-43e5-81c5-665aca7b2787 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:35.595791 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1780110-dda7-4202-a3bc-e5569945e899 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:35.601793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-0d035905-98c8-43e5-81c5-665aca7b2787 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:35.603849 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0d035905-98c8-43e5-81c5-665aca7b2787 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:35.638187 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2077bf30-be00-43ef-91ef-e4b12e65f996 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:35.638811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2077bf30-be00-43ef-91ef-e4b12e65f996 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:35.639124 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2077bf30-be00-43ef-91ef-e4b12e65f996 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:35.639396 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2077bf30-be00-43ef-91ef-e4b12e65f996 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:35.641046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2077bf30-be00-43ef-91ef-e4b12e65f996 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:35.642066 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2077bf30-be00-43ef-91ef-e4b12e65f996 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:35.678344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 20/41] 10.222.0.15 () {60 vars in 1141 bytes} [Tue Sep 13 01:04:35 2022] POST /identity/v3/projects => generated 301 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:04:35.679010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2077bf30-be00-43ef-91ef-e4b12e65f996 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:35.716108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 22/42] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Sep 13 01:04:35.773620 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:35.811142 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2077bf30-be00-43ef-91ef-e4b12e65f996 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:35.838576 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d035905-98c8-43e5-81c5-665aca7b2787 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:35.876769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773bff41-d018-4136-80aa-54a05a0bc0f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:35.877323 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773bff41-d018-4136-80aa-54a05a0bc0f2 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:35.877652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773bff41-d018-4136-80aa-54a05a0bc0f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:35.877930 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773bff41-d018-4136-80aa-54a05a0bc0f2 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:35.878615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773bff41-d018-4136-80aa-54a05a0bc0f2 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:35.880201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773bff41-d018-4136-80aa-54a05a0bc0f2 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:35.899290 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 21/43] 10.222.0.15 () {60 vars in 1131 bytes} [Tue Sep 13 01:04:35 2022] POST /identity/v3/roles => generated 225 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:04:35.963910 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d035905-98c8-43e5-81c5-665aca7b2787 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:35.984708 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:36.005213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881164dd-9d45-48d1-adfa-04157b38626d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:36.005901 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881164dd-9d45-48d1-adfa-04157b38626d None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:36.006124 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881164dd-9d45-48d1-adfa-04157b38626d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:36.006396 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881164dd-9d45-48d1-adfa-04157b38626d None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:36.007117 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881164dd-9d45-48d1-adfa-04157b38626d None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:36.008862 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773bff41-d018-4136-80aa-54a05a0bc0f2 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:36.011948 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881164dd-9d45-48d1-adfa-04157b38626d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:36.143710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773bff41-d018-4136-80aa-54a05a0bc0f2 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:36.179549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:36.189395 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6101b0-61bb-4ec4-a65f-84ed1eb5023d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:36.190430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6101b0-61bb-4ec4-a65f-84ed1eb5023d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:36.190990 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6101b0-61bb-4ec4-a65f-84ed1eb5023d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:36.191479 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6101b0-61bb-4ec4-a65f-84ed1eb5023d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:36.193659 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6101b0-61bb-4ec4-a65f-84ed1eb5023d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:36.195184 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6101b0-61bb-4ec4-a65f-84ed1eb5023d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:36.197116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 22/44] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:04:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:36.202950 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6101b0-61bb-4ec4-a65f-84ed1eb5023d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:36.262579 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 23/45] 10.222.0.15 () {60 vars in 1131 bytes} [Tue Sep 13 01:04:35 2022] POST /identity/v3/roles => generated 223 bytes in 424 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:04:36.317322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6101b0-61bb-4ec4-a65f-84ed1eb5023d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:36.354498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30e1d4a-4738-4f22-9083-310ca9a834c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:36.355168 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30e1d4a-4738-4f22-9083-310ca9a834c0 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:36.355455 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30e1d4a-4738-4f22-9083-310ca9a834c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:36.355733 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30e1d4a-4738-4f22-9083-310ca9a834c0 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:36.356403 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30e1d4a-4738-4f22-9083-310ca9a834c0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:36.357870 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30e1d4a-4738-4f22-9083-310ca9a834c0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:36.391037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 23/46] 10.222.0.15 () {60 vars in 1141 bytes} [Tue Sep 13 01:04:36 2022] POST /identity/v3/projects => generated 312 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:04:37.905244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc55cf90-a87e-4899-afe1-02d16ec0ee95 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:37.905758 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc55cf90-a87e-4899-afe1-02d16ec0ee95 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:37.905839 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc55cf90-a87e-4899-afe1-02d16ec0ee95 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:37.907172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 24/47] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:37.912291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f175ca-9689-4cf7-96a4-221c4fbb6dec None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:37.912717 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f175ca-9689-4cf7-96a4-221c4fbb6dec None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:37.912848 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f175ca-9689-4cf7-96a4-221c4fbb6dec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:37.931738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-f6f175ca-9689-4cf7-96a4-221c4fbb6dec None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:37.932918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f6f175ca-9689-4cf7-96a4-221c4fbb6dec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:37.977802 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:37.999392 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 24/48] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:38.264190 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc55cf90-a87e-4899-afe1-02d16ec0ee95 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:38.364822 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc55cf90-a87e-4899-afe1-02d16ec0ee95 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:38.397963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42c2000-dd2a-493f-9c96-e68244dfb24d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:38.398593 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42c2000-dd2a-493f-9c96-e68244dfb24d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:38.398900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42c2000-dd2a-493f-9c96-e68244dfb24d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:38.399181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42c2000-dd2a-493f-9c96-e68244dfb24d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:38.413563 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42c2000-dd2a-493f-9c96-e68244dfb24d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:04:38.418731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-a42c2000-dd2a-493f-9c96-e68244dfb24d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:04:38.418731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:04:38.418731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:04:38.418731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:04:38.418731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:04:38.418731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:04:38.418731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:04:38.418731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:04:38.418731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:04:38.418731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:04:38.418731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:04:38.418731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:04:38.418731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:04:38.418731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:04:38.418731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:04:38.418731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:04:38.419792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:04:38.419792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:04:38.419792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:04:38.419792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:04:38.419792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:04:38.419792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:04:38.419792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:04:38.419792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:04:38.419792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:04:38.419792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:04:38.419792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:04:38.419792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:04:38.419792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:04:38.419792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:04:38.419792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:04:38.419792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:04:38.420722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:04:38.420722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:04:38.420722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:04:38.420722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:04:38.420722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:04:38.420722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:04:38.420722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:04:38.420722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:04:38.420722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:04:38.420722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:04:38.420722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:04:38.420722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:04:38.420722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 25/49] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:04:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:04:38.426081 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f175ca-9689-4cf7-96a4-221c4fbb6dec None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:38.527212 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f175ca-9689-4cf7-96a4-221c4fbb6dec None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:38.533489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb4c9b2-f9bf-4f46-aba6-fd8dcc89270d None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:38.533774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb4c9b2-f9bf-4f46-aba6-fd8dcc89270d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:38.534049 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb4c9b2-f9bf-4f46-aba6-fd8dcc89270d None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:38.535458 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 26/50] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:04:38.540510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b49685-df8c-4199-b0be-e9f4fbdd9010 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:38.540792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b49685-df8c-4199-b0be-e9f4fbdd9010 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:38.541073 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b49685-df8c-4199-b0be-e9f4fbdd9010 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:38.559362 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f2b49685-df8c-4199-b0be-e9f4fbdd9010 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:38.560882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f2b49685-df8c-4199-b0be-e9f4fbdd9010 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:38.561260 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df3db49-8550-4e72-8fde-0b163fbd903d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:38.561871 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df3db49-8550-4e72-8fde-0b163fbd903d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:38.562415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df3db49-8550-4e72-8fde-0b163fbd903d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:38.562684 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df3db49-8550-4e72-8fde-0b163fbd903d None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:38.563372 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df3db49-8550-4e72-8fde-0b163fbd903d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:38.563671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df3db49-8550-4e72-8fde-0b163fbd903d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:38.565039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df3db49-8550-4e72-8fde-0b163fbd903d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:38.573730 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 25/51] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:04:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:38.579321 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df3db49-8550-4e72-8fde-0b163fbd903d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:38.589303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:38.634800 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 27/52] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 01:04:38.724858 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:38.780183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df3db49-8550-4e72-8fde-0b163fbd903d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:38.812431 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405b3138-a5fc-4611-8f1d-15598fb9e3db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:38.813056 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405b3138-a5fc-4611-8f1d-15598fb9e3db None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:38.813333 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405b3138-a5fc-4611-8f1d-15598fb9e3db None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:38.813606 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405b3138-a5fc-4611-8f1d-15598fb9e3db None admin] PATH_INFO: `/v3/users/22ddb5151c0f4696a4f99a9acc8804ea` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:38.814712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405b3138-a5fc-4611-8f1d-15598fb9e3db None admin] RBAC: Authorizing `identity:get_user(user_id=22ddb5151c0f4696a4f99a9acc8804ea)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:38.816163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405b3138-a5fc-4611-8f1d-15598fb9e3db None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:38.817715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 26/53] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:04:38 2022] GET /identity/v3/users/22ddb5151c0f4696a4f99a9acc8804ea => generated 247 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:38.823831 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b49685-df8c-4199-b0be-e9f4fbdd9010 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:38.903408 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405b3138-a5fc-4611-8f1d-15598fb9e3db None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:38.919391 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b49685-df8c-4199-b0be-e9f4fbdd9010 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:38.953534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a080857-5f52-4d15-84cd-2753210acb25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:38.954154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a080857-5f52-4d15-84cd-2753210acb25 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:38.954434 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a080857-5f52-4d15-84cd-2753210acb25 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:38.954702 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a080857-5f52-4d15-84cd-2753210acb25 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:38.955823 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a080857-5f52-4d15-84cd-2753210acb25 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:38.957393 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a080857-5f52-4d15-84cd-2753210acb25 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:38.959106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 28/54] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:04:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:38.964784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a080857-5f52-4d15-84cd-2753210acb25 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:38.993990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:39.002120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405b3138-a5fc-4611-8f1d-15598fb9e3db None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:39.013299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:39.037155 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40db5fb-b1d0-469c-afca-6608656c4b7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:39.037716 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40db5fb-b1d0-469c-afca-6608656c4b7e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:39.037992 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40db5fb-b1d0-469c-afca-6608656c4b7e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:39.038261 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40db5fb-b1d0-469c-afca-6608656c4b7e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:39.039380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40db5fb-b1d0-469c-afca-6608656c4b7e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:39.040874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40db5fb-b1d0-469c-afca-6608656c4b7e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:39.042410 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 27/55] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:04:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:39.047775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40db5fb-b1d0-469c-afca-6608656c4b7e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:39.064544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a080857-5f52-4d15-84cd-2753210acb25 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:39.097064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28e0709-c43f-4e94-bec0-0cd820f97652 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:39.097662 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28e0709-c43f-4e94-bec0-0cd820f97652 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:39.097939 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28e0709-c43f-4e94-bec0-0cd820f97652 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:39.098212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28e0709-c43f-4e94-bec0-0cd820f97652 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:39.098965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28e0709-c43f-4e94-bec0-0cd820f97652 None admin] RBAC: Adding query filter params (role.id=66a36c858aab40688080eed6ceace3a0, scope.domain.id=default, user.id=22ddb5151c0f4696a4f99a9acc8804ea) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:39.099275 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28e0709-c43f-4e94-bec0-0cd820f97652 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:39.100715 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28e0709-c43f-4e94-bec0-0cd820f97652 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:39.106894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 29/56] 10.222.0.15 () {56 vars in 1322 bytes} [Tue Sep 13 01:04:38 2022] GET /identity/v3/role_assignments?user.id=22ddb5151c0f4696a4f99a9acc8804ea&scope.domain.id=default&role.id=66a36c858aab40688080eed6ceace3a0 => generated 530 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:39.152894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40db5fb-b1d0-469c-afca-6608656c4b7e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:39.189488 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebfa874-0333-438e-9785-07fb180c01aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:39.190114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebfa874-0333-438e-9785-07fb180c01aa None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:39.190396 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebfa874-0333-438e-9785-07fb180c01aa None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:39.190670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebfa874-0333-438e-9785-07fb180c01aa None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:39.191373 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebfa874-0333-438e-9785-07fb180c01aa None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:39.192827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebfa874-0333-438e-9785-07fb180c01aa None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:39.221411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 28/57] 10.222.0.15 () {60 vars in 1140 bytes} [Tue Sep 13 01:04:39 2022] POST /identity/v3/projects => generated 298 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:04:41.354299 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd61ec7-14cd-4283-a3d2-ccf932f24bf0 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:41.354817 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd61ec7-14cd-4283-a3d2-ccf932f24bf0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:41.354956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd61ec7-14cd-4283-a3d2-ccf932f24bf0 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:41.356229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 30/58] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:41.361282 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7a386a-6f03-475f-9e89-7baec0dcc6ec None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:41.361683 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7a386a-6f03-475f-9e89-7baec0dcc6ec None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:41.361833 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7a386a-6f03-475f-9e89-7baec0dcc6ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:41.380018 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ea7a386a-6f03-475f-9e89-7baec0dcc6ec None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:41.381190 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ea7a386a-6f03-475f-9e89-7baec0dcc6ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:41.403622 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:41.446325 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 29/59] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 01:04:41.710152 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd61ec7-14cd-4283-a3d2-ccf932f24bf0 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:41.802381 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:41.810058 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd61ec7-14cd-4283-a3d2-ccf932f24bf0 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:41.842403 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fc8fb7-5f37-4d9e-8040-d4d138e55be2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:41.843045 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fc8fb7-5f37-4d9e-8040-d4d138e55be2 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:41.843330 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fc8fb7-5f37-4d9e-8040-d4d138e55be2 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:41.843605 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fc8fb7-5f37-4d9e-8040-d4d138e55be2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:41.844709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fc8fb7-5f37-4d9e-8040-d4d138e55be2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:41.846126 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fc8fb7-5f37-4d9e-8040-d4d138e55be2 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:41.847681 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 31/60] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:04:41 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:41.853565 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7a386a-6f03-475f-9e89-7baec0dcc6ec None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:41.946833 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7a386a-6f03-475f-9e89-7baec0dcc6ec None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:41.979578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cf6c84-6297-4aa0-a260-8152ac301e59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:41.980178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cf6c84-6297-4aa0-a260-8152ac301e59 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:41.980459 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cf6c84-6297-4aa0-a260-8152ac301e59 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:41.980734 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cf6c84-6297-4aa0-a260-8152ac301e59 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:41.981374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cf6c84-6297-4aa0-a260-8152ac301e59 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:41.982808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cf6c84-6297-4aa0-a260-8152ac301e59 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:42.009299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 30/61] 10.222.0.15 () {60 vars in 1132 bytes} [Tue Sep 13 01:04:41 2022] POST /identity/v3/users => generated 275 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:04:45.331776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f07f90-ac67-4649-93a3-1a28fbacb20d None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:45.332790 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f07f90-ac67-4649-93a3-1a28fbacb20d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:45.333294 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f07f90-ac67-4649-93a3-1a28fbacb20d None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:45.334847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 32/62] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:45.340925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cb8090-42ea-4afe-9ae3-c646feaa32c0 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:45.341457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cb8090-42ea-4afe-9ae3-c646feaa32c0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:45.341457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cb8090-42ea-4afe-9ae3-c646feaa32c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:45.366392 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-02cb8090-42ea-4afe-9ae3-c646feaa32c0 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:45.371894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-02cb8090-42ea-4afe-9ae3-c646feaa32c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:45.381565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101a8923-abe8-41de-b0aa-79cff233aa6f None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:45.382208 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101a8923-abe8-41de-b0aa-79cff233aa6f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:45.382682 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101a8923-abe8-41de-b0aa-79cff233aa6f None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:45.385159 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:45.386317 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 33/63] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:45 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:45.389970 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3366fa-d2ce-42a1-9973-5f5a484c995b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:45.390507 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3366fa-d2ce-42a1-9973-5f5a484c995b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:45.391015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3366fa-d2ce-42a1-9973-5f5a484c995b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:45.426800 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-5c3366fa-d2ce-42a1-9973-5f5a484c995b None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:45.428041 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5c3366fa-d2ce-42a1-9973-5f5a484c995b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:45.459329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 31/64] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:45.501499 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb8d5a5-1df3-49bf-aa5b-cf1fb665cfec None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:45.502111 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb8d5a5-1df3-49bf-aa5b-cf1fb665cfec None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:45.502587 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb8d5a5-1df3-49bf-aa5b-cf1fb665cfec None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:45.504204 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 32/65] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:04:45.511264 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365e636f-c12a-4e71-b59d-0109d729cc3c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:45.511829 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365e636f-c12a-4e71-b59d-0109d729cc3c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:45.512302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365e636f-c12a-4e71-b59d-0109d729cc3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:45.519207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 34/66] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 01:04:45.535293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-365e636f-c12a-4e71-b59d-0109d729cc3c None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:45.536746 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-365e636f-c12a-4e71-b59d-0109d729cc3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:45.620265 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 33/67] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:45.752763 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3366fa-d2ce-42a1-9973-5f5a484c995b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:45.877810 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:45.877970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365e636f-c12a-4e71-b59d-0109d729cc3c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:45.896485 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3366fa-d2ce-42a1-9973-5f5a484c995b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:45.938157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba87e40-3e5f-48e3-b497-544eb002a8fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:45.938796 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba87e40-3e5f-48e3-b497-544eb002a8fe None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:45.939110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba87e40-3e5f-48e3-b497-544eb002a8fe None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:45.939384 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba87e40-3e5f-48e3-b497-544eb002a8fe None admin] PATH_INFO: `/v3/roles/member` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:45.954928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba87e40-3e5f-48e3-b497-544eb002a8fe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:04:45.961181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-8ba87e40-3e5f-48e3-b497-544eb002a8fe None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:04:45.961181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:04:45.961181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:04:45.961181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:04:45.961181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:04:45.961181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:04:45.961181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:04:45.961181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:04:45.961181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:04:45.961181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:04:45.961181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:04:45.961181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:04:45.961181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:04:45.961181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:04:45.961181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:04:45.961181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:04:45.961991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:04:45.961991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:04:45.961991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:04:45.961991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:04:45.961991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:04:45.961991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:04:45.961991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:04:45.961991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:04:45.961991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:04:45.961991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:04:45.961991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:04:45.961991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:04:45.961991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:04:45.961991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:04:45.961991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:04:45.961991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:04:45.962872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:04:45.962872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:04:45.962872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:04:45.962872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:04:45.962872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:04:45.962872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:04:45.962872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:04:45.962872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:04:45.962872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:04:45.962872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:04:45.962872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:04:45.962872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:04:45.964660 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 35/68] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:04:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:04:45.966161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4e6493-7302-4ac2-971a-b0914ad2e9f9 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:45.966446 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4e6493-7302-4ac2-971a-b0914ad2e9f9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:45.966725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4e6493-7302-4ac2-971a-b0914ad2e9f9 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:45.967996 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 36/69] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:45.973010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4e6493-7302-4ac2-971a-b0914ad2e9f9 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:45.975224 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:45.999068 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365e636f-c12a-4e71-b59d-0109d729cc3c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:46.037179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ef6a5a-45c1-4685-9356-6f5f5991f627 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:46.037816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ef6a5a-45c1-4685-9356-6f5f5991f627 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:46.038095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ef6a5a-45c1-4685-9356-6f5f5991f627 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:46.038363 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ef6a5a-45c1-4685-9356-6f5f5991f627 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:46.049801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ef6a5a-45c1-4685-9356-6f5f5991f627 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:04:46.056226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-66ef6a5a-45c1-4685-9356-6f5f5991f627 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:04:46.056226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:04:46.056226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:04:46.056226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:04:46.056226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:04:46.056226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:04:46.056226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:04:46.056226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:04:46.056226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:04:46.056226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:04:46.056226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:04:46.056226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:04:46.056226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:04:46.056226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:04:46.056226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:04:46.056226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:04:46.057057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:04:46.057057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:04:46.057057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:04:46.057057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:04:46.057057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:04:46.057057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:04:46.057057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:04:46.057057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:04:46.057057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:04:46.057057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:04:46.057057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:04:46.057057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:04:46.057057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:04:46.057057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:04:46.057057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:04:46.057057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:04:46.057805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:04:46.057805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:04:46.057805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:04:46.057805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:04:46.057805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:04:46.057805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:04:46.057805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:04:46.057805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:04:46.057805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:04:46.057805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:04:46.057805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:04:46.057805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:04:46.057805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 34/70] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:04:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:04:46.058434 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02371b54-ed4d-4b0c-9e69-20bbd10c0563 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:46.059133 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02371b54-ed4d-4b0c-9e69-20bbd10c0563 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:46.059133 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02371b54-ed4d-4b0c-9e69-20bbd10c0563 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:46.077851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-02371b54-ed4d-4b0c-9e69-20bbd10c0563 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:46.079062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-02371b54-ed4d-4b0c-9e69-20bbd10c0563 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:46.082053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4e6493-7302-4ac2-971a-b0914ad2e9f9 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:46.114702 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6106e6-1e77-4d23-9015-d9d8d0c0d8dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:46.115328 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6106e6-1e77-4d23-9015-d9d8d0c0d8dd None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:46.115610 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6106e6-1e77-4d23-9015-d9d8d0c0d8dd None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:46.115883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6106e6-1e77-4d23-9015-d9d8d0c0d8dd None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:46.116562 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6106e6-1e77-4d23-9015-d9d8d0c0d8dd None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:46.116865 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6106e6-1e77-4d23-9015-d9d8d0c0d8dd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:46.118241 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6106e6-1e77-4d23-9015-d9d8d0c0d8dd None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:46.126891 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 37/71] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:04:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:46.127897 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd1bd9b-49f6-4100-90b9-9cc45a493596 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:46.128129 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd1bd9b-49f6-4100-90b9-9cc45a493596 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:46.128406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd1bd9b-49f6-4100-90b9-9cc45a493596 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:46.129595 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 38/72] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:46 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:46.130285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd1bd9b-49f6-4100-90b9-9cc45a493596 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:46.146526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:46.152635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 35/73] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:46.153479 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02371b54-ed4d-4b0c-9e69-20bbd10c0563 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:46.222883 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:46.244813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd1bd9b-49f6-4100-90b9-9cc45a493596 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:46.260465 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02371b54-ed4d-4b0c-9e69-20bbd10c0563 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:46.279377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cdb532-0a76-4c40-8365-a9660faef7a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:46.279982 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cdb532-0a76-4c40-8365-a9660faef7a1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:46.280266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cdb532-0a76-4c40-8365-a9660faef7a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:46.280539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cdb532-0a76-4c40-8365-a9660faef7a1 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:46.281194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cdb532-0a76-4c40-8365-a9660faef7a1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:46.281495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cdb532-0a76-4c40-8365-a9660faef7a1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:46.282899 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cdb532-0a76-4c40-8365-a9660faef7a1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:46.291393 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 39/74] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:04:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:46.296971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7439c77-174a-46bb-9cf4-5f224ea7ad88 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:46.297265 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7439c77-174a-46bb-9cf4-5f224ea7ad88 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:46.297542 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7439c77-174a-46bb-9cf4-5f224ea7ad88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:46.298187 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2d0bbe-7c91-4a5b-8b12-58d4a28f4291 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:46.299047 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2d0bbe-7c91-4a5b-8b12-58d4a28f4291 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:46.299504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2d0bbe-7c91-4a5b-8b12-58d4a28f4291 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:46.299910 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2d0bbe-7c91-4a5b-8b12-58d4a28f4291 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:46.312129 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2d0bbe-7c91-4a5b-8b12-58d4a28f4291 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:04:46.317215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e7439c77-174a-46bb-9cf4-5f224ea7ad88 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:46.317322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-6c2d0bbe-7c91-4a5b-8b12-58d4a28f4291 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:04:46.317322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:04:46.317322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:04:46.317322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:04:46.317322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:04:46.317322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:04:46.317322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:04:46.317322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:04:46.317322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:04:46.317322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:04:46.317322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:04:46.317322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:04:46.317322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:04:46.317322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:04:46.317322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:04:46.317322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:04:46.318134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:04:46.318134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:04:46.318134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:04:46.318134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:04:46.318134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:04:46.318134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:04:46.318134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:04:46.318134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:04:46.318134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:04:46.318134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:04:46.318134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:04:46.318134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:04:46.318134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:04:46.318134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:04:46.318134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:04:46.318134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:04:46.323820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:04:46.323820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:04:46.323820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:04:46.323820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:04:46.323820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:04:46.323820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:04:46.323820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:04:46.323820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:04:46.323820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:04:46.323820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:04:46.323820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:04:46.323820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:04:46.323820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 36/75] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:04:46 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:04:46.324471 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e7439c77-174a-46bb-9cf4-5f224ea7ad88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:46.324538 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2d0bbe-7c91-4a5b-8b12-58d4a28f4291 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:46.389826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 40/76] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:46.391741 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7439c77-174a-46bb-9cf4-5f224ea7ad88 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:46.403376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:46.445469 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2d0bbe-7c91-4a5b-8b12-58d4a28f4291 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:46.474088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:46.484106 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e258b2-1738-467c-b6c3-1da83e28be9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:46.484738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e258b2-1738-467c-b6c3-1da83e28be9f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:46.485019 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e258b2-1738-467c-b6c3-1da83e28be9f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:46.485296 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e258b2-1738-467c-b6c3-1da83e28be9f None admin] PATH_INFO: `/v3/users/1e80dcbdbef640078cfec6d97eabca95` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:46.503028 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e258b2-1738-467c-b6c3-1da83e28be9f None admin] RBAC: Authorizing `identity:get_user(user_id=1e80dcbdbef640078cfec6d97eabca95)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:46.504500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e258b2-1738-467c-b6c3-1da83e28be9f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:46.506769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 37/77] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:04:46 2022] GET /identity/v3/users/1e80dcbdbef640078cfec6d97eabca95 => generated 275 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:46.507544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e258b2-1738-467c-b6c3-1da83e28be9f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:46.524456 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7439c77-174a-46bb-9cf4-5f224ea7ad88 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:46.562809 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5941ca6-3978-4e72-8cb8-9c692ff38aea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:46.563560 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5941ca6-3978-4e72-8cb8-9c692ff38aea None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:46.563847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5941ca6-3978-4e72-8cb8-9c692ff38aea None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:46.564127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5941ca6-3978-4e72-8cb8-9c692ff38aea None admin] PATH_INFO: `/v3/users/22ddb5151c0f4696a4f99a9acc8804ea` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:46.565266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5941ca6-3978-4e72-8cb8-9c692ff38aea None admin] RBAC: Authorizing `identity:get_user(user_id=22ddb5151c0f4696a4f99a9acc8804ea)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:46.566718 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5941ca6-3978-4e72-8cb8-9c692ff38aea None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:46.568667 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 41/78] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:04:46 2022] GET /identity/v3/users/22ddb5151c0f4696a4f99a9acc8804ea => generated 247 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:46.569069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5941ca6-3978-4e72-8cb8-9c692ff38aea None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:46.617466 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e258b2-1738-467c-b6c3-1da83e28be9f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:46.650466 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:46.651744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea11b31-d8e3-43f2-8df1-a05ac88a5625 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:46.652327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea11b31-d8e3-43f2-8df1-a05ac88a5625 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:46.652702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea11b31-d8e3-43f2-8df1-a05ac88a5625 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:46.653070 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea11b31-d8e3-43f2-8df1-a05ac88a5625 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:46.653733 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea11b31-d8e3-43f2-8df1-a05ac88a5625 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:46.654038 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea11b31-d8e3-43f2-8df1-a05ac88a5625 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:46.655434 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea11b31-d8e3-43f2-8df1-a05ac88a5625 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:46.663677 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 38/79] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:04:46 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:46.664463 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea11b31-d8e3-43f2-8df1-a05ac88a5625 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:46.681515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5941ca6-3978-4e72-8cb8-9c692ff38aea None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:46.714557 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600a5aeb-e2c0-47f7-97fc-53a4cead8f87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:46.715204 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600a5aeb-e2c0-47f7-97fc-53a4cead8f87 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:46.715486 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600a5aeb-e2c0-47f7-97fc-53a4cead8f87 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:46.715634 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:46.715750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600a5aeb-e2c0-47f7-97fc-53a4cead8f87 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:46.716857 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600a5aeb-e2c0-47f7-97fc-53a4cead8f87 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:46.718515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600a5aeb-e2c0-47f7-97fc-53a4cead8f87 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:46.720099 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 42/80] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:04:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:46.720859 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600a5aeb-e2c0-47f7-97fc-53a4cead8f87 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:46.769312 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea11b31-d8e3-43f2-8df1-a05ac88a5625 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:46.802076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1064d1c6-06a6-44b4-9212-4352b304bf8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:46.802680 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1064d1c6-06a6-44b4-9212-4352b304bf8e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:46.802988 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1064d1c6-06a6-44b4-9212-4352b304bf8e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:46.803261 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1064d1c6-06a6-44b4-9212-4352b304bf8e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:46.819225 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600a5aeb-e2c0-47f7-97fc-53a4cead8f87 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:46.853852 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df75eafc-9021-495d-8aaf-488830467901 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:46.854454 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df75eafc-9021-495d-8aaf-488830467901 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:46.854740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df75eafc-9021-495d-8aaf-488830467901 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:46.855055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df75eafc-9021-495d-8aaf-488830467901 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:46.866424 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:46.870342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1064d1c6-06a6-44b4-9212-4352b304bf8e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:46.871864 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1064d1c6-06a6-44b4-9212-4352b304bf8e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:46.917035 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:46.921912 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df75eafc-9021-495d-8aaf-488830467901 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:46.923411 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df75eafc-9021-495d-8aaf-488830467901 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:46.970294 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 39/81] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:04:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:04:46.971139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1064d1c6-06a6-44b4-9212-4352b304bf8e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:47.021656 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 43/82] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:04:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:04:47.022481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df75eafc-9021-495d-8aaf-488830467901 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:47.065746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:47.068408 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1064d1c6-06a6-44b4-9212-4352b304bf8e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:47.104227 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63192a9-f570-4695-b939-7df51d2e5762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:47.105029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63192a9-f570-4695-b939-7df51d2e5762 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:47.105310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63192a9-f570-4695-b939-7df51d2e5762 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:47.105579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63192a9-f570-4695-b939-7df51d2e5762 None admin] PATH_INFO: `/v3/users/1e80dcbdbef640078cfec6d97eabca95` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:47.106698 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63192a9-f570-4695-b939-7df51d2e5762 None admin] RBAC: Authorizing `identity:get_user(user_id=1e80dcbdbef640078cfec6d97eabca95)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:47.108167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63192a9-f570-4695-b939-7df51d2e5762 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:47.109724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 40/83] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:04:46 2022] GET /identity/v3/users/1e80dcbdbef640078cfec6d97eabca95 => generated 275 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:47.110460 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63192a9-f570-4695-b939-7df51d2e5762 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:47.124982 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:47.126517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df75eafc-9021-495d-8aaf-488830467901 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:47.160572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94cf51d-8f87-4348-a4ee-779de77f3b60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:47.161205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94cf51d-8f87-4348-a4ee-779de77f3b60 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:47.161490 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94cf51d-8f87-4348-a4ee-779de77f3b60 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:47.161771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94cf51d-8f87-4348-a4ee-779de77f3b60 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:47.172219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94cf51d-8f87-4348-a4ee-779de77f3b60 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:04:47.176927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-a94cf51d-8f87-4348-a4ee-779de77f3b60 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:04:47.176927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:04:47.176927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:04:47.176927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:04:47.176927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:04:47.176927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:04:47.176927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:04:47.176927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:04:47.176927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:04:47.176927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:04:47.176927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:04:47.176927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:04:47.176927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:04:47.176927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:04:47.176927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:04:47.176927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:04:47.177819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:04:47.177819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:04:47.177819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:04:47.177819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:04:47.177819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:04:47.177819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:04:47.177819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:04:47.177819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:04:47.177819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:04:47.177819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:04:47.177819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:04:47.177819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:04:47.177819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:04:47.177819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:04:47.177819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:04:47.177819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:04:47.178565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:04:47.178565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:04:47.178565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:04:47.178565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:04:47.178565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:04:47.178565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:04:47.178565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:04:47.178565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:04:47.178565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:04:47.178565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:04:47.178565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:04:47.178565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:04:47.178565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 44/84] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:04:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:04:47.179252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94cf51d-8f87-4348-a4ee-779de77f3b60 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:47.208346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63192a9-f570-4695-b939-7df51d2e5762 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:47.242750 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485abbf8-f279-438e-8b8d-fe23c6825e54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:47.243382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485abbf8-f279-438e-8b8d-fe23c6825e54 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:47.243733 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485abbf8-f279-438e-8b8d-fe23c6825e54 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:47.243931 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485abbf8-f279-438e-8b8d-fe23c6825e54 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:47.244573 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485abbf8-f279-438e-8b8d-fe23c6825e54 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:47.246000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485abbf8-f279-438e-8b8d-fe23c6825e54 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:47.267192 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 41/85] 10.222.0.15 () {60 vars in 1132 bytes} [Tue Sep 13 01:04:47 2022] POST /identity/v3/users => generated 289 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:04:47.267997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485abbf8-f279-438e-8b8d-fe23c6825e54 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:47.283880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94cf51d-8f87-4348-a4ee-779de77f3b60 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:47.285689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:47.325215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbed2bf-abc6-4c5f-9dd7-05643a1fdca1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:47.325885 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbed2bf-abc6-4c5f-9dd7-05643a1fdca1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:47.326184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbed2bf-abc6-4c5f-9dd7-05643a1fdca1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:47.326453 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbed2bf-abc6-4c5f-9dd7-05643a1fdca1 None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:47.327722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bbed2bf-abc6-4c5f-9dd7-05643a1fdca1 None admin] RBAC: Authorizing `identity:get_project(project_id=4426c2e7285b4cbc8331aa86690ef664)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:47.329254 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bbed2bf-abc6-4c5f-9dd7-05643a1fdca1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:47.336240 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 45/86] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:04:47 2022] GET /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664 => generated 298 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:47.336240 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbed2bf-abc6-4c5f-9dd7-05643a1fdca1 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:47.357847 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:47.398147 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485abbf8-f279-438e-8b8d-fe23c6825e54 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:47.441793 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36827af-53da-44d4-b755-c19d876659a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:47.442384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36827af-53da-44d4-b755-c19d876659a5 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:47.442666 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36827af-53da-44d4-b755-c19d876659a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:47.442972 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36827af-53da-44d4-b755-c19d876659a5 None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:47.444117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36827af-53da-44d4-b755-c19d876659a5 None admin] RBAC: Authorizing `identity:get_project(project_id=4426c2e7285b4cbc8331aa86690ef664)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:47.445550 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36827af-53da-44d4-b755-c19d876659a5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:47.447275 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 42/87] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:04:47 2022] GET /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664 => generated 298 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:47.448037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36827af-53da-44d4-b755-c19d876659a5 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:47.455182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbed2bf-abc6-4c5f-9dd7-05643a1fdca1 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:47.494093 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9210ba78-76a3-4111-bd9d-619371ec22ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:47.496552 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9210ba78-76a3-4111-bd9d-619371ec22ac None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:47.496552 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9210ba78-76a3-4111-bd9d-619371ec22ac None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:47.496552 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9210ba78-76a3-4111-bd9d-619371ec22ac None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:47.540163 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:47.581626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36827af-53da-44d4-b755-c19d876659a5 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:47.587748 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9210ba78-76a3-4111-bd9d-619371ec22ac None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:47.589438 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9210ba78-76a3-4111-bd9d-619371ec22ac None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:47.624479 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960bd05c-f6fd-4a85-bd2f-f6699a820609 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:47.625893 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960bd05c-f6fd-4a85-bd2f-f6699a820609 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:47.626320 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960bd05c-f6fd-4a85-bd2f-f6699a820609 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:47.626733 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960bd05c-f6fd-4a85-bd2f-f6699a820609 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:47.628476 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-960bd05c-f6fd-4a85-bd2f-f6699a820609 None admin] RBAC: Adding query filter params (name=member) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:47.628947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-960bd05c-f6fd-4a85-bd2f-f6699a820609 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:47.631233 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-960bd05c-f6fd-4a85-bd2f-f6699a820609 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:47.637684 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:47.644055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 43/88] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:04:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:47.645429 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960bd05c-f6fd-4a85-bd2f-f6699a820609 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:47.712709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 46/89] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:04:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:04:47.718302 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9210ba78-76a3-4111-bd9d-619371ec22ac None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:47.782092 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960bd05c-f6fd-4a85-bd2f-f6699a820609 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:47.796292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:47.828268 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c909c7-2d5d-4153-a1a2-48859699a5d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:47.829165 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c909c7-2d5d-4153-a1a2-48859699a5d4 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:47.829444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c909c7-2d5d-4153-a1a2-48859699a5d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:47.829724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c909c7-2d5d-4153-a1a2-48859699a5d4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:47.830900 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c909c7-2d5d-4153-a1a2-48859699a5d4 None admin] RBAC: Adding query filter params (role.id=6f71034643cb4e8caee299f5d0bb863f, scope.project.id=4426c2e7285b4cbc8331aa86690ef664, user.id=1e80dcbdbef640078cfec6d97eabca95) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:47.831222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c909c7-2d5d-4153-a1a2-48859699a5d4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:47.833232 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c909c7-2d5d-4153-a1a2-48859699a5d4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:47.837832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9210ba78-76a3-4111-bd9d-619371ec22ac None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:47.843672 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 44/90] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:04:47 2022] GET /identity/v3/role_assignments?user.id=1e80dcbdbef640078cfec6d97eabca95&scope.project.id=4426c2e7285b4cbc8331aa86690ef664&role.id=6f71034643cb4e8caee299f5d0bb863f => generated 259 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:47.844300 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c909c7-2d5d-4153-a1a2-48859699a5d4 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:47.900804 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283198ca-3f3c-44d6-b170-6098ff697bab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:47.901849 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283198ca-3f3c-44d6-b170-6098ff697bab None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:47.902146 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283198ca-3f3c-44d6-b170-6098ff697bab None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:47.902404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283198ca-3f3c-44d6-b170-6098ff697bab None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:47.903224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283198ca-3f3c-44d6-b170-6098ff697bab None admin] RBAC: Adding query filter params (role.id=66a36c858aab40688080eed6ceace3a0, scope.project.id=4426c2e7285b4cbc8331aa86690ef664, user.id=22ddb5151c0f4696a4f99a9acc8804ea) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:47.903662 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283198ca-3f3c-44d6-b170-6098ff697bab None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:47.905276 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283198ca-3f3c-44d6-b170-6098ff697bab None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:47.911852 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 47/91] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:04:47 2022] GET /identity/v3/role_assignments?user.id=22ddb5151c0f4696a4f99a9acc8804ea&scope.project.id=4426c2e7285b4cbc8331aa86690ef664&role.id=66a36c858aab40688080eed6ceace3a0 => generated 259 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:47.912711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283198ca-3f3c-44d6-b170-6098ff697bab None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:48.007047 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:48.076898 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c909c7-2d5d-4153-a1a2-48859699a5d4 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:48.140817 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283198ca-3f3c-44d6-b170-6098ff697bab None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:48.146123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8cddce-6e5d-4e05-81d5-64024b149476 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:48.146783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8cddce-6e5d-4e05-81d5-64024b149476 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:48.147086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8cddce-6e5d-4e05-81d5-64024b149476 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:48.147366 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8cddce-6e5d-4e05-81d5-64024b149476 None admin] PATH_INFO: `/v3/users/1e80dcbdbef640078cfec6d97eabca95` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:48.153274 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8cddce-6e5d-4e05-81d5-64024b149476 None admin] RBAC: Authorizing `identity:get_user(user_id=1e80dcbdbef640078cfec6d97eabca95)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:48.154768 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8cddce-6e5d-4e05-81d5-64024b149476 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:48.156700 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 45/92] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:04:47 2022] GET /identity/v3/users/1e80dcbdbef640078cfec6d97eabca95 => generated 275 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:48.163974 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8cddce-6e5d-4e05-81d5-64024b149476 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:48.230828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5c112c-7779-4f6f-a515-75877dd3b439 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:48.231492 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5c112c-7779-4f6f-a515-75877dd3b439 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:48.231823 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5c112c-7779-4f6f-a515-75877dd3b439 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:48.234894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5c112c-7779-4f6f-a515-75877dd3b439 None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:48.234894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5c112c-7779-4f6f-a515-75877dd3b439 None admin] RBAC: Authorizing `identity:get_project(project_id=4426c2e7285b4cbc8331aa86690ef664)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:48.235668 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5c112c-7779-4f6f-a515-75877dd3b439 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:48.237563 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 48/93] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:04:47 2022] GET /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664 => generated 298 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:48.244459 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5c112c-7779-4f6f-a515-75877dd3b439 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:48.250824 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:48.326261 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8cddce-6e5d-4e05-81d5-64024b149476 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:48.360548 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:48.374018 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0486a5-6072-43ac-a67a-4e9209a139a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:48.374665 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0486a5-6072-43ac-a67a-4e9209a139a4 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:48.374979 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0486a5-6072-43ac-a67a-4e9209a139a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:48.375258 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0486a5-6072-43ac-a67a-4e9209a139a4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:48.471155 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5c112c-7779-4f6f-a515-75877dd3b439 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:48.497910 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0486a5-6072-43ac-a67a-4e9209a139a4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:48.499022 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0486a5-6072-43ac-a67a-4e9209a139a4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:48.575594 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a11af5-fb35-4a4c-9122-cc028a199c01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:48.576230 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a11af5-fb35-4a4c-9122-cc028a199c01 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:48.576564 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a11af5-fb35-4a4c-9122-cc028a199c01 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:48.576799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a11af5-fb35-4a4c-9122-cc028a199c01 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:48.577530 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a11af5-fb35-4a4c-9122-cc028a199c01 None admin] RBAC: Adding query filter params (role.id=cd616e816a394068867091116a3f1ce0, scope.project.id=4426c2e7285b4cbc8331aa86690ef664, user.id=1e80dcbdbef640078cfec6d97eabca95) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:48.577821 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a11af5-fb35-4a4c-9122-cc028a199c01 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:48.579287 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a11af5-fb35-4a4c-9122-cc028a199c01 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:48.597070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 49/94] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:04:48 2022] GET /identity/v3/role_assignments?user.id=1e80dcbdbef640078cfec6d97eabca95&scope.project.id=4426c2e7285b4cbc8331aa86690ef664&role.id=cd616e816a394068867091116a3f1ce0 => generated 259 bytes in 353 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:48.664046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 46/95] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:04:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 498 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:04:48.667855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a11af5-fb35-4a4c-9122-cc028a199c01 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:48.755872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:48.801183 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a11af5-fb35-4a4c-9122-cc028a199c01 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:48.846329 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6773db3c-363d-435c-bc46-b43549f9286d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:48.846915 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6773db3c-363d-435c-bc46-b43549f9286d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:48.847200 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6773db3c-363d-435c-bc46-b43549f9286d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:48.847478 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6773db3c-363d-435c-bc46-b43549f9286d None admin] PATH_INFO: `/v3/projects/bf2e886952254e57a265bbc1fb8be1a1` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:48.849311 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6773db3c-363d-435c-bc46-b43549f9286d None admin] RBAC: Authorizing `identity:get_project(project_id=bf2e886952254e57a265bbc1fb8be1a1)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:48.850781 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6773db3c-363d-435c-bc46-b43549f9286d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:48.853259 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 50/96] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:04:48 2022] GET /identity/v3/projects/bf2e886952254e57a265bbc1fb8be1a1 => generated 312 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:48.859194 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0486a5-6072-43ac-a67a-4e9209a139a4 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:48.908665 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:48.996275 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0486a5-6072-43ac-a67a-4e9209a139a4 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:49.032555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7980afe-d71e-45f0-9ac5-76743f72f9da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:49.033503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7980afe-d71e-45f0-9ac5-76743f72f9da None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:49.033989 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7980afe-d71e-45f0-9ac5-76743f72f9da None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:49.034528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7980afe-d71e-45f0-9ac5-76743f72f9da None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:49.035484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7980afe-d71e-45f0-9ac5-76743f72f9da None admin] RBAC: Adding query filter params (role.id=6f71034643cb4e8caee299f5d0bb863f, scope.project.id=bf2e886952254e57a265bbc1fb8be1a1, user.id=1e80dcbdbef640078cfec6d97eabca95) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:49.036013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7980afe-d71e-45f0-9ac5-76743f72f9da None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:49.037638 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7980afe-d71e-45f0-9ac5-76743f72f9da None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:49.043904 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 47/97] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:04:48 2022] GET /identity/v3/role_assignments?user.id=1e80dcbdbef640078cfec6d97eabca95&scope.project.id=bf2e886952254e57a265bbc1fb8be1a1&role.id=6f71034643cb4e8caee299f5d0bb863f => generated 259 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:50.016146 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb739e81-7e5c-4d1f-9018-305931c7e40a None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:50.017169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb739e81-7e5c-4d1f-9018-305931c7e40a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:50.017660 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb739e81-7e5c-4d1f-9018-305931c7e40a None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:50.019410 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 51/98] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:50.025941 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9491343d-48fb-41f2-ba3a-4956759e7c56 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:50.026802 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9491343d-48fb-41f2-ba3a-4956759e7c56 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:50.027310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9491343d-48fb-41f2-ba3a-4956759e7c56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:50.051881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9491343d-48fb-41f2-ba3a-4956759e7c56 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:50.053588 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9491343d-48fb-41f2-ba3a-4956759e7c56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:50.140815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 48/99] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:50.528291 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867972b4-7d6b-4c2f-a45b-5c52e3dec458 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:50.529120 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867972b4-7d6b-4c2f-a45b-5c52e3dec458 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:50.529684 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867972b4-7d6b-4c2f-a45b-5c52e3dec458 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:50.531273 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 52/100] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:50.536527 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784a1e2f-738c-4d8d-a030-99719f1a9486 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:50.537044 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784a1e2f-738c-4d8d-a030-99719f1a9486 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:50.537521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784a1e2f-738c-4d8d-a030-99719f1a9486 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:50.561806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-784a1e2f-738c-4d8d-a030-99719f1a9486 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:50.563573 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-784a1e2f-738c-4d8d-a030-99719f1a9486 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:50.592660 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:50.645984 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 49/101] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:50.745441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867972b4-7d6b-4c2f-a45b-5c52e3dec458 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:50.881351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867972b4-7d6b-4c2f-a45b-5c52e3dec458 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:50.930134 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746b1645-d4a3-42ab-9843-9087d3aa9c01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:50.931363 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746b1645-d4a3-42ab-9843-9087d3aa9c01 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:50.931648 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746b1645-d4a3-42ab-9843-9087d3aa9c01 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:50.931921 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746b1645-d4a3-42ab-9843-9087d3aa9c01 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:50.933100 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746b1645-d4a3-42ab-9843-9087d3aa9c01 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:50.934567 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746b1645-d4a3-42ab-9843-9087d3aa9c01 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:50.936425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 53/102] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:04:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:50.944610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784a1e2f-738c-4d8d-a030-99719f1a9486 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:51.028365 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45ee473-afea-4619-9e33-5c17a115b38b None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:51.028844 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45ee473-afea-4619-9e33-5c17a115b38b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:51.028844 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45ee473-afea-4619-9e33-5c17a115b38b None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:51.030171 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 54/103] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:51.041398 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72dcee48-962b-4bc5-8881-ec9c9a1079a5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:51.041684 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72dcee48-962b-4bc5-8881-ec9c9a1079a5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:51.041960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72dcee48-962b-4bc5-8881-ec9c9a1079a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:51.054902 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:51.072391 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-72dcee48-962b-4bc5-8881-ec9c9a1079a5 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:51.073893 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-72dcee48-962b-4bc5-8881-ec9c9a1079a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:51.105692 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784a1e2f-738c-4d8d-a030-99719f1a9486 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:51.148807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62bce16-ecae-45a1-9bf9-b269f57342e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:51.149429 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62bce16-ecae-45a1-9bf9-b269f57342e0 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:51.149772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62bce16-ecae-45a1-9bf9-b269f57342e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:51.150049 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62bce16-ecae-45a1-9bf9-b269f57342e0 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:51.150707 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62bce16-ecae-45a1-9bf9-b269f57342e0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:51.152204 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62bce16-ecae-45a1-9bf9-b269f57342e0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:51.189894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 50/104] 10.222.0.15 () {60 vars in 1141 bytes} [Tue Sep 13 01:04:50 2022] POST /identity/v3/projects => generated 302 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:04:51.205027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 55/105] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:51.278081 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62bce16-ecae-45a1-9bf9-b269f57342e0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:51.291431 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:51.397825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62bce16-ecae-45a1-9bf9-b269f57342e0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:51.448467 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d76a35-5e45-4535-9351-e11835dd9c6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:51.449139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d76a35-5e45-4535-9351-e11835dd9c6a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:51.449424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d76a35-5e45-4535-9351-e11835dd9c6a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:51.449702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d76a35-5e45-4535-9351-e11835dd9c6a None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:51.462686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d76a35-5e45-4535-9351-e11835dd9c6a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:04:51.468059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-32d76a35-5e45-4535-9351-e11835dd9c6a None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 01:04:51.468059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:04:51.468059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:04:51.468059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:04:51.468059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:04:51.468059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:04:51.468059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:04:51.468059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:04:51.468059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:04:51.468059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:04:51.468059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:04:51.468059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:04:51.468059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:04:51.468059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:04:51.468059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:04:51.468059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:04:51.468874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:04:51.468874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:04:51.468874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:04:51.468874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:04:51.468874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:04:51.468874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:04:51.468874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:04:51.468874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:04:51.468874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:04:51.468874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:04:51.468874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:04:51.468874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:04:51.468874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:04:51.468874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:04:51.468874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:04:51.468874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:04:51.469635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:04:51.469635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:04:51.469635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:04:51.469635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:04:51.469635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:04:51.469635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:04:51.469635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:04:51.469635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:04:51.469635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:04:51.469635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:04:51.469635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 01:04:51.469635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:04:51.469635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 51/106] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:04:51 2022] GET /identity/v3/roles/reader => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:04:51.475141 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72dcee48-962b-4bc5-8881-ec9c9a1079a5 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:51.540971 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cebd4d4-2fee-4094-b442-6049eeb93751 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:51.541312 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cebd4d4-2fee-4094-b442-6049eeb93751 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:51.541585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cebd4d4-2fee-4094-b442-6049eeb93751 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:51.543130 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 52/107] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:04:51.548458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e88338-d39e-49a7-bc1f-4226f601178e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:51.548808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e88338-d39e-49a7-bc1f-4226f601178e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:51.549087 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e88338-d39e-49a7-bc1f-4226f601178e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:51.568549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-13e88338-d39e-49a7-bc1f-4226f601178e None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:51.569819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-13e88338-d39e-49a7-bc1f-4226f601178e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:51.619358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72dcee48-962b-4bc5-8881-ec9c9a1079a5 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:51.638669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:51.645712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 53/108] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 01:04:51.646680 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e88338-d39e-49a7-bc1f-4226f601178e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:51.660836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:51.668211 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c386ff-0ac6-4033-a0be-206ede47b0ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:51.668840 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c386ff-0ac6-4033-a0be-206ede47b0ea None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:51.669123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c386ff-0ac6-4033-a0be-206ede47b0ea None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:51.669398 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c386ff-0ac6-4033-a0be-206ede47b0ea None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:51.670072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c386ff-0ac6-4033-a0be-206ede47b0ea None admin] RBAC: Adding query filter params (name=reader) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:51.670379 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c386ff-0ac6-4033-a0be-206ede47b0ea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:51.671788 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c386ff-0ac6-4033-a0be-206ede47b0ea None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:51.684578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 56/109] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:04:51 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:51.699907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c386ff-0ac6-4033-a0be-206ede47b0ea None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:51.770088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e88338-d39e-49a7-bc1f-4226f601178e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:51.806596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8392a4e8-10db-45a3-be06-eb2bc14330d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:51.807296 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8392a4e8-10db-45a3-be06-eb2bc14330d8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:51.807636 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8392a4e8-10db-45a3-be06-eb2bc14330d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:51.807914 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8392a4e8-10db-45a3-be06-eb2bc14330d8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:51.819641 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8392a4e8-10db-45a3-be06-eb2bc14330d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:04:51.825253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-8392a4e8-10db-45a3-be06-eb2bc14330d8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:04:51.825253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:04:51.825253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:04:51.825253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:04:51.825253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:04:51.825253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:04:51.825253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:04:51.825253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:04:51.825253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:04:51.825253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:04:51.825253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:04:51.825253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:04:51.825253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:04:51.825253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:04:51.825253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:04:51.825253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:04:51.826107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:04:51.826107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:04:51.826107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:04:51.826107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:04:51.826107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:04:51.826107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:04:51.826107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:04:51.826107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:04:51.826107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:04:51.826107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:04:51.826107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:04:51.826107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:04:51.826107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:04:51.826107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:04:51.826107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:04:51.826107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:04:51.826880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:04:51.826880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:04:51.826880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:04:51.826880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:04:51.826880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:04:51.826880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:04:51.826880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:04:51.826880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:04:51.826880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:04:51.826880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:04:51.826880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:04:51.826880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:04:51.826880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 54/110] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:04:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:04:51.838018 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8392a4e8-10db-45a3-be06-eb2bc14330d8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:51.854650 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c386ff-0ac6-4033-a0be-206ede47b0ea None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:51.897898 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79b9f92-74a3-4997-94f8-676c76fe87b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:51.898543 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79b9f92-74a3-4997-94f8-676c76fe87b2 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:51.898669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:51.898832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79b9f92-74a3-4997-94f8-676c76fe87b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:51.899116 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79b9f92-74a3-4997-94f8-676c76fe87b2 None admin] PATH_INFO: `/v3/users/f7c6f07cdd5f4461ae915f6a9d72b7ca` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:51.919203 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79b9f92-74a3-4997-94f8-676c76fe87b2 None admin] RBAC: Authorizing `identity:get_user(user_id=f7c6f07cdd5f4461ae915f6a9d72b7ca)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:51.920725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79b9f92-74a3-4997-94f8-676c76fe87b2 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:51.922524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 57/111] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:04:51 2022] GET /identity/v3/users/f7c6f07cdd5f4461ae915f6a9d72b7ca => generated 289 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:51.928137 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79b9f92-74a3-4997-94f8-676c76fe87b2 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:51.962965 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8392a4e8-10db-45a3-be06-eb2bc14330d8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:52.014828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:52.063672 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4a7ec2-2a63-4091-a66d-6e83b4b9215e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:52.064228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4a7ec2-2a63-4091-a66d-6e83b4b9215e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:52.064512 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4a7ec2-2a63-4091-a66d-6e83b4b9215e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:52.064789 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4a7ec2-2a63-4091-a66d-6e83b4b9215e None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:52.065472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4a7ec2-2a63-4091-a66d-6e83b4b9215e None admin] RBAC: Adding query filter params (name=member) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:52.065778 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4a7ec2-2a63-4091-a66d-6e83b4b9215e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:52.067204 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4a7ec2-2a63-4091-a66d-6e83b4b9215e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:52.085995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 55/112] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:04:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:52.091703 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4a7ec2-2a63-4091-a66d-6e83b4b9215e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:52.118401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79b9f92-74a3-4997-94f8-676c76fe87b2 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:52.175661 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ba6bc2-a0a6-4890-b52e-659439a76ac7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:52.176297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ba6bc2-a0a6-4890-b52e-659439a76ac7 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:52.176581 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ba6bc2-a0a6-4890-b52e-659439a76ac7 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:52.176916 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ba6bc2-a0a6-4890-b52e-659439a76ac7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:52.266630 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4a7ec2-2a63-4091-a66d-6e83b4b9215e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:52.272029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ba6bc2-a0a6-4890-b52e-659439a76ac7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:52.273839 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ba6bc2-a0a6-4890-b52e-659439a76ac7 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:52.290256 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:52.308684 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6119f746-1de5-4293-a8e9-6ae3e92636c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:52.309334 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6119f746-1de5-4293-a8e9-6ae3e92636c1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:52.309624 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6119f746-1de5-4293-a8e9-6ae3e92636c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:52.309906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6119f746-1de5-4293-a8e9-6ae3e92636c1 None admin] PATH_INFO: `/v3/users/1e80dcbdbef640078cfec6d97eabca95` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:52.311105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6119f746-1de5-4293-a8e9-6ae3e92636c1 None admin] RBAC: Authorizing `identity:get_user(user_id=1e80dcbdbef640078cfec6d97eabca95)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:52.313255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6119f746-1de5-4293-a8e9-6ae3e92636c1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:52.314928 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 56/113] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:04:52 2022] GET /identity/v3/users/1e80dcbdbef640078cfec6d97eabca95 => generated 275 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:52.315635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6119f746-1de5-4293-a8e9-6ae3e92636c1 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:52.337138 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:52.387597 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 58/114] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:04:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 460 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 13 01:04:52.388982 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1893dfd3-e1cf-497a-a0d5-f41d730d08d2 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:52.389596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1893dfd3-e1cf-497a-a0d5-f41d730d08d2 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:52.389596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1893dfd3-e1cf-497a-a0d5-f41d730d08d2 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:52.390766 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 59/115] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:52.391792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9813e2f2-a1e9-4f66-a154-8439cfacb6e0 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:52.392070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9813e2f2-a1e9-4f66-a154-8439cfacb6e0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:52.392451 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9813e2f2-a1e9-4f66-a154-8439cfacb6e0 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:52.393627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 60/116] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:52 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:52.394309 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9813e2f2-a1e9-4f66-a154-8439cfacb6e0 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:52.435168 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6119f746-1de5-4293-a8e9-6ae3e92636c1 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:52.472072 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3c4bc7-db4a-45c4-939a-e9bc134ea978 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:52.472720 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3c4bc7-db4a-45c4-939a-e9bc134ea978 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:52.473004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3c4bc7-db4a-45c4-939a-e9bc134ea978 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:52.473282 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3c4bc7-db4a-45c4-939a-e9bc134ea978 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:52.486443 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3c4bc7-db4a-45c4-939a-e9bc134ea978 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:04:52.492179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-fe3c4bc7-db4a-45c4-939a-e9bc134ea978 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:04:52.492179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:04:52.492179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:04:52.492179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:04:52.492179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:04:52.492179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:04:52.492179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:04:52.492179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:04:52.492179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:04:52.492179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:04:52.492179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:04:52.492179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:04:52.492179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:04:52.492179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:04:52.492179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:04:52.492179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:04:52.493098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:04:52.493098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:04:52.493098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:04:52.493098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:04:52.493098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:04:52.493098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:04:52.493098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:04:52.493098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:04:52.493098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:04:52.493098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:04:52.493098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:04:52.493098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:04:52.493098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:04:52.493098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:04:52.493098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:04:52.493098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:04:52.495803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:04:52.495803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:04:52.495803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:04:52.495803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:04:52.495803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:04:52.495803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:04:52.495803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:04:52.495803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:04:52.495803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:04:52.495803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:04:52.495803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:04:52.495803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:04:52.495803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 57/117] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:04:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:04:52.495803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3c4bc7-db4a-45c4-939a-e9bc134ea978 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:52.508393 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9813e2f2-a1e9-4f66-a154-8439cfacb6e0 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:52.546425 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:52.557315 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ee804c-850a-413b-a610-3688f6ef9454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:52.558843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ee804c-850a-413b-a610-3688f6ef9454 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:52.559481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ee804c-850a-413b-a610-3688f6ef9454 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:52.560009 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ee804c-850a-413b-a610-3688f6ef9454 None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:52.561879 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ee804c-850a-413b-a610-3688f6ef9454 None admin] RBAC: Authorizing `identity:get_project(project_id=4426c2e7285b4cbc8331aa86690ef664)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:52.564063 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ee804c-850a-413b-a610-3688f6ef9454 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:52.565955 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 61/118] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:04:52 2022] GET /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664 => generated 298 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:52.572018 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaeffc2-b23b-404c-adf0-54ca4ce6dc7f None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:52.572307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaeffc2-b23b-404c-adf0-54ca4ce6dc7f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:52.572587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaeffc2-b23b-404c-adf0-54ca4ce6dc7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:52.591340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-bdaeffc2-b23b-404c-adf0-54ca4ce6dc7f None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:52.592768 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-bdaeffc2-b23b-404c-adf0-54ca4ce6dc7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:52.603830 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3c4bc7-db4a-45c4-939a-e9bc134ea978 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:52.608077 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:52.641341 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4c1a09-a5af-4764-acee-43dbef07492b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:52.641973 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4c1a09-a5af-4764-acee-43dbef07492b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:52.642261 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4c1a09-a5af-4764-acee-43dbef07492b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:52.642530 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4c1a09-a5af-4764-acee-43dbef07492b None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:52.643675 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4c1a09-a5af-4764-acee-43dbef07492b None admin] RBAC: Authorizing `identity:get_project(project_id=4426c2e7285b4cbc8331aa86690ef664)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:52.645275 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4c1a09-a5af-4764-acee-43dbef07492b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:52.652095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 58/119] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:04:52 2022] GET /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664 => generated 298 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:52.652095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d994cc66-2f72-46a5-8419-98cb43b2e734 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:52.652095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d994cc66-2f72-46a5-8419-98cb43b2e734 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:52.652095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d994cc66-2f72-46a5-8419-98cb43b2e734 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:52.670885 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 62/120] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:52.670885 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdaeffc2-b23b-404c-adf0-54ca4ce6dc7f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:52.678197 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-d994cc66-2f72-46a5-8419-98cb43b2e734 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:52.679508 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d994cc66-2f72-46a5-8419-98cb43b2e734 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:52.756712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 59/121] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:52.757679 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d994cc66-2f72-46a5-8419-98cb43b2e734 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:52.767500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdaeffc2-b23b-404c-adf0-54ca4ce6dc7f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:52.836686 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be983288-aa27-4630-b10c-0ee08b350020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:52.837960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be983288-aa27-4630-b10c-0ee08b350020 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:52.838355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be983288-aa27-4630-b10c-0ee08b350020 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:52.838721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be983288-aa27-4630-b10c-0ee08b350020 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:52.839493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be983288-aa27-4630-b10c-0ee08b350020 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:52.840060 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be983288-aa27-4630-b10c-0ee08b350020 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:52.843192 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:52.843828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be983288-aa27-4630-b10c-0ee08b350020 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:52.858262 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 63/122] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:04:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:52.859718 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be983288-aa27-4630-b10c-0ee08b350020 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:52.890613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:52.928963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d994cc66-2f72-46a5-8419-98cb43b2e734 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:53.000709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f984b6ae-2d26-4916-bd40-bcb317596d3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:53.001589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f984b6ae-2d26-4916-bd40-bcb317596d3f None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:53.001967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f984b6ae-2d26-4916-bd40-bcb317596d3f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:53.002339 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f984b6ae-2d26-4916-bd40-bcb317596d3f None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664/users/1e80dcbdbef640078cfec6d97eabca95/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:53.007647 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f984b6ae-2d26-4916-bd40-bcb317596d3f None admin] RBAC: Authorizing `identity:create_grant(project_id=4426c2e7285b4cbc8331aa86690ef664, user_id=1e80dcbdbef640078cfec6d97eabca95, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:53.009313 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f984b6ae-2d26-4916-bd40-bcb317596d3f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:53.025797 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 60/123] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:04:52 2022] PUT /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664/users/1e80dcbdbef640078cfec6d97eabca95/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:04:53.027211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f984b6ae-2d26-4916-bd40-bcb317596d3f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:53.032143 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be983288-aa27-4630-b10c-0ee08b350020 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:53.079068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f57077-b7b2-4b81-99b4-ee013fb4b923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:53.080372 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f57077-b7b2-4b81-99b4-ee013fb4b923 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:53.080659 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f57077-b7b2-4b81-99b4-ee013fb4b923 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:53.080937 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f57077-b7b2-4b81-99b4-ee013fb4b923 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:53.081749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f57077-b7b2-4b81-99b4-ee013fb4b923 None admin] RBAC: Adding query filter params (role.id=6f98018cf28149aa84201e6dc4a43c2d, scope.project.id=4426c2e7285b4cbc8331aa86690ef664, user.id=f7c6f07cdd5f4461ae915f6a9d72b7ca) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:53.082065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f57077-b7b2-4b81-99b4-ee013fb4b923 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:53.084103 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f57077-b7b2-4b81-99b4-ee013fb4b923 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:53.093678 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 64/124] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:04:52 2022] GET /identity/v3/role_assignments?user.id=f7c6f07cdd5f4461ae915f6a9d72b7ca&scope.project.id=4426c2e7285b4cbc8331aa86690ef664&role.id=6f98018cf28149aa84201e6dc4a43c2d => generated 259 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:53.094556 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f57077-b7b2-4b81-99b4-ee013fb4b923 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:53.162350 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f984b6ae-2d26-4916-bd40-bcb317596d3f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:53.178654 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:53.219769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d581b82-f2a0-4e35-889e-eed225ec6edb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:53.220422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d581b82-f2a0-4e35-889e-eed225ec6edb None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:53.220991 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d581b82-f2a0-4e35-889e-eed225ec6edb None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:53.221252 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d581b82-f2a0-4e35-889e-eed225ec6edb None admin] PATH_INFO: `/v3/users/22ddb5151c0f4696a4f99a9acc8804ea` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:53.222987 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d581b82-f2a0-4e35-889e-eed225ec6edb None admin] RBAC: Authorizing `identity:get_user(user_id=22ddb5151c0f4696a4f99a9acc8804ea)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:53.224666 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d581b82-f2a0-4e35-889e-eed225ec6edb None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:53.226665 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 61/125] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:04:53 2022] GET /identity/v3/users/22ddb5151c0f4696a4f99a9acc8804ea => generated 247 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:53.227657 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d581b82-f2a0-4e35-889e-eed225ec6edb None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:53.257243 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:53.314394 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f57077-b7b2-4b81-99b4-ee013fb4b923 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:53.354149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d581b82-f2a0-4e35-889e-eed225ec6edb None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:53.365559 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf06a81b-3771-42f0-b526-2dd87450f795 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:53.366205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf06a81b-3771-42f0-b526-2dd87450f795 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:53.366489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf06a81b-3771-42f0-b526-2dd87450f795 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:53.366769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf06a81b-3771-42f0-b526-2dd87450f795 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:53.382235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf06a81b-3771-42f0-b526-2dd87450f795 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:04:53.390173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27a7690-7f3e-4979-ac48-6771f87c00ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:53.390534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-bf06a81b-3771-42f0-b526-2dd87450f795 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:04:53.390534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:04:53.390534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:04:53.390534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:04:53.390534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:04:53.390534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:04:53.390534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:04:53.390534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:04:53.390534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:04:53.390534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:04:53.390534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:04:53.390534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:04:53.390534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:04:53.390534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:04:53.390534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:04:53.390534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:04:53.391407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:04:53.391407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:04:53.391407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:04:53.391407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:04:53.391407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:04:53.391407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:04:53.391407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:04:53.391407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:04:53.391407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:04:53.391407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:04:53.391407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:04:53.391407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:04:53.391407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:04:53.391407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:04:53.391407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:04:53.391407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:04:53.392184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:04:53.392184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:04:53.392184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:04:53.392184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:04:53.392184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:04:53.392184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:04:53.392184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:04:53.392184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:04:53.392184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:04:53.392184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:04:53.392184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:04:53.392184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:04:53.392759 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27a7690-7f3e-4979-ac48-6771f87c00ff None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:53.392759 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27a7690-7f3e-4979-ac48-6771f87c00ff None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:53.392759 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27a7690-7f3e-4979-ac48-6771f87c00ff None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:53.394350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 65/126] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:04:53 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:04:53.395480 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf06a81b-3771-42f0-b526-2dd87450f795 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:53.411266 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27a7690-7f3e-4979-ac48-6771f87c00ff None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:04:53.417651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-c27a7690-7f3e-4979-ac48-6771f87c00ff None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:04:53.417651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:04:53.417651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:04:53.417651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:04:53.417651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:04:53.417651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:04:53.417651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:04:53.417651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:04:53.417651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:04:53.417651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:04:53.417651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:04:53.417651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:04:53.417651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:04:53.417651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:04:53.417651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:04:53.417651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:04:53.418544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:04:53.418544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:04:53.418544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:04:53.418544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:04:53.418544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:04:53.418544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:04:53.418544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:04:53.418544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:04:53.418544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:04:53.418544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:04:53.418544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:04:53.418544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:04:53.418544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:04:53.418544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:04:53.418544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:04:53.418544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:04:53.419515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:04:53.419515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:04:53.419515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:04:53.419515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:04:53.419515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:04:53.419515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:04:53.419515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:04:53.419515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:04:53.419515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:04:53.419515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:04:53.419515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:04:53.419515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:04:53.422120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 62/127] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:04:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:04:53.423162 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27a7690-7f3e-4979-ac48-6771f87c00ff None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:53.463354 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:53.515758 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf06a81b-3771-42f0-b526-2dd87450f795 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:53.542259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27a7690-7f3e-4979-ac48-6771f87c00ff None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:53.551014 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29a9309-0639-400f-815d-9cfcea322823 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:53.552093 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29a9309-0639-400f-815d-9cfcea322823 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:53.552093 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29a9309-0639-400f-815d-9cfcea322823 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:53.552287 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29a9309-0639-400f-815d-9cfcea322823 None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:53.553473 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29a9309-0639-400f-815d-9cfcea322823 None admin] RBAC: Authorizing `identity:get_project(project_id=4426c2e7285b4cbc8331aa86690ef664)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:53.561347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29a9309-0639-400f-815d-9cfcea322823 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:53.561347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 66/128] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:04:53 2022] GET /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664 => generated 298 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:53.561347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29a9309-0639-400f-815d-9cfcea322823 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:53.567230 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:53.596054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fefc300-699a-4d2e-acd2-ca673c7b105c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:53.596673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fefc300-699a-4d2e-acd2-ca673c7b105c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:53.596952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fefc300-699a-4d2e-acd2-ca673c7b105c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:53.597226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fefc300-699a-4d2e-acd2-ca673c7b105c None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:53.597891 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fefc300-699a-4d2e-acd2-ca673c7b105c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:53.598193 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fefc300-699a-4d2e-acd2-ca673c7b105c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:53.599824 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fefc300-699a-4d2e-acd2-ca673c7b105c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:53.609403 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 63/129] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:04:53 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:53.610049 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fefc300-699a-4d2e-acd2-ca673c7b105c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:53.718836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fefc300-699a-4d2e-acd2-ca673c7b105c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:53.727322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:53.764316 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9515ab14-1245-42a1-8d19-32c327cdc567 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:53.765122 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9515ab14-1245-42a1-8d19-32c327cdc567 None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:53.765399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9515ab14-1245-42a1-8d19-32c327cdc567 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:53.765670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9515ab14-1245-42a1-8d19-32c327cdc567 None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664/users/22ddb5151c0f4696a4f99a9acc8804ea/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:53.768843 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9515ab14-1245-42a1-8d19-32c327cdc567 None admin] RBAC: Authorizing `identity:create_grant(project_id=4426c2e7285b4cbc8331aa86690ef664, user_id=22ddb5151c0f4696a4f99a9acc8804ea, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:53.770286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9515ab14-1245-42a1-8d19-32c327cdc567 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:53.789556 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 64/130] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:04:53 2022] PUT /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664/users/22ddb5151c0f4696a4f99a9acc8804ea/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:04:53.790587 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9515ab14-1245-42a1-8d19-32c327cdc567 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:53.870709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29a9309-0639-400f-815d-9cfcea322823 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:53.984808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ea4310-e2cb-4469-8fd4-3218e9f191c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:53.985804 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ea4310-e2cb-4469-8fd4-3218e9f191c0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:53.986284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ea4310-e2cb-4469-8fd4-3218e9f191c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:53.986752 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ea4310-e2cb-4469-8fd4-3218e9f191c0 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:53.987675 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ea4310-e2cb-4469-8fd4-3218e9f191c0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:53.988244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ea4310-e2cb-4469-8fd4-3218e9f191c0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:53.989829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ea4310-e2cb-4469-8fd4-3218e9f191c0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:54.006640 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 67/131] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:04:53 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 449 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:54.012105 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ea4310-e2cb-4469-8fd4-3218e9f191c0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:54.020134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9515ab14-1245-42a1-8d19-32c327cdc567 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:54.084274 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7e6a97-b56b-4a0e-958d-268a0e3ff6a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:54.085307 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7e6a97-b56b-4a0e-958d-268a0e3ff6a2 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:54.085588 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7e6a97-b56b-4a0e-958d-268a0e3ff6a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:54.085860 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7e6a97-b56b-4a0e-958d-268a0e3ff6a2 None admin] PATH_INFO: `/v3/users/1e80dcbdbef640078cfec6d97eabca95` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:54.090356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7e6a97-b56b-4a0e-958d-268a0e3ff6a2 None admin] RBAC: Authorizing `identity:get_user(user_id=1e80dcbdbef640078cfec6d97eabca95)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:54.092132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7e6a97-b56b-4a0e-958d-268a0e3ff6a2 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:54.093843 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 65/132] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:04:53 2022] GET /identity/v3/users/1e80dcbdbef640078cfec6d97eabca95 => generated 275 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:54.108270 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7e6a97-b56b-4a0e-958d-268a0e3ff6a2 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:54.152173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:54.168672 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:54.219766 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ea4310-e2cb-4469-8fd4-3218e9f191c0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:54.256862 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6120af28-1bac-4961-98e6-a630ebf12bd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:54.257509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6120af28-1bac-4961-98e6-a630ebf12bd7 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:54.258026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6120af28-1bac-4961-98e6-a630ebf12bd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:54.258026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6120af28-1bac-4961-98e6-a630ebf12bd7 None admin] PATH_INFO: `/v3/users/1e80dcbdbef640078cfec6d97eabca95` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:54.259746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6120af28-1bac-4961-98e6-a630ebf12bd7 None admin] RBAC: Authorizing `identity:get_user(user_id=1e80dcbdbef640078cfec6d97eabca95)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:54.261209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6120af28-1bac-4961-98e6-a630ebf12bd7 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:54.263144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 68/133] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:04:54 2022] GET /identity/v3/users/1e80dcbdbef640078cfec6d97eabca95 => generated 275 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:54.268972 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7e6a97-b56b-4a0e-958d-268a0e3ff6a2 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:54.271959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6120af28-1bac-4961-98e6-a630ebf12bd7 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:54.362987 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537699ae-bdbb-42fb-98f2-fc80c723f20a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:54.364334 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537699ae-bdbb-42fb-98f2-fc80c723f20a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:54.365524 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537699ae-bdbb-42fb-98f2-fc80c723f20a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:54.366002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537699ae-bdbb-42fb-98f2-fc80c723f20a None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:54.368696 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537699ae-bdbb-42fb-98f2-fc80c723f20a None admin] RBAC: Authorizing `identity:get_project(project_id=4426c2e7285b4cbc8331aa86690ef664)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:54.370361 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537699ae-bdbb-42fb-98f2-fc80c723f20a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:54.372982 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 66/134] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:04:54 2022] GET /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664 => generated 298 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:54.377247 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d7e223-3c07-4d9b-8bad-a0a7408b0c24 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:54.378703 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d7e223-3c07-4d9b-8bad-a0a7408b0c24 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:54.379364 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d7e223-3c07-4d9b-8bad-a0a7408b0c24 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:54.381614 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 67/135] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:54 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:54.382829 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d7e223-3c07-4d9b-8bad-a0a7408b0c24 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:54.479354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6120af28-1bac-4961-98e6-a630ebf12bd7 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:54.536428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:54.541020 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ee244b-b15d-4e5a-92f8-a6220052633b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:54.541661 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ee244b-b15d-4e5a-92f8-a6220052633b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:54.541941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ee244b-b15d-4e5a-92f8-a6220052633b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:54.542216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ee244b-b15d-4e5a-92f8-a6220052633b None admin] PATH_INFO: `/v3/projects/bf2e886952254e57a265bbc1fb8be1a1` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:54.547626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ee244b-b15d-4e5a-92f8-a6220052633b None admin] RBAC: Authorizing `identity:get_project(project_id=bf2e886952254e57a265bbc1fb8be1a1)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:54.549104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ee244b-b15d-4e5a-92f8-a6220052633b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:54.549462 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d7e223-3c07-4d9b-8bad-a0a7408b0c24 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:54.551116 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 69/136] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:04:54 2022] GET /identity/v3/projects/bf2e886952254e57a265bbc1fb8be1a1 => generated 312 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:54.552273 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605ab189-a092-48d0-8c45-966e2e0503c0 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:54.552554 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605ab189-a092-48d0-8c45-966e2e0503c0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:54.552827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605ab189-a092-48d0-8c45-966e2e0503c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:54.566184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:54.585966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-605ab189-a092-48d0-8c45-966e2e0503c0 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:54.589946 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-605ab189-a092-48d0-8c45-966e2e0503c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:54.608640 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07998513-7b1d-4321-821b-a426fd2aaecb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:54.609627 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07998513-7b1d-4321-821b-a426fd2aaecb None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:54.610134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07998513-7b1d-4321-821b-a426fd2aaecb None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:54.610624 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07998513-7b1d-4321-821b-a426fd2aaecb None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664/users/1e80dcbdbef640078cfec6d97eabca95/roles/cd616e816a394068867091116a3f1ce0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:54.613141 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07998513-7b1d-4321-821b-a426fd2aaecb None admin] RBAC: Authorizing `identity:create_grant(project_id=4426c2e7285b4cbc8331aa86690ef664, user_id=1e80dcbdbef640078cfec6d97eabca95, role_id=cd616e816a394068867091116a3f1ce0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:54.614783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07998513-7b1d-4321-821b-a426fd2aaecb None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:54.640798 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 68/137] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:04:54 2022] PUT /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664/users/1e80dcbdbef640078cfec6d97eabca95/roles/cd616e816a394068867091116a3f1ce0 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:04:54.641601 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07998513-7b1d-4321-821b-a426fd2aaecb None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:54.723723 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 70/138] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:54.794695 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07998513-7b1d-4321-821b-a426fd2aaecb None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:54.832673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766b92b9-9976-4f11-ac90-1236a17955bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:54.833349 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766b92b9-9976-4f11-ac90-1236a17955bd None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:54.833628 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766b92b9-9976-4f11-ac90-1236a17955bd None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:54.833909 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766b92b9-9976-4f11-ac90-1236a17955bd None admin] PATH_INFO: `/v3/projects/bf2e886952254e57a265bbc1fb8be1a1/users/1e80dcbdbef640078cfec6d97eabca95/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:54.835869 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766b92b9-9976-4f11-ac90-1236a17955bd None admin] RBAC: Authorizing `identity:create_grant(project_id=bf2e886952254e57a265bbc1fb8be1a1, user_id=1e80dcbdbef640078cfec6d97eabca95, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:54.837352 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766b92b9-9976-4f11-ac90-1236a17955bd None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:54.853430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 69/139] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:04:54 2022] PUT /identity/v3/projects/bf2e886952254e57a265bbc1fb8be1a1/users/1e80dcbdbef640078cfec6d97eabca95/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:04:55.181026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605ab189-a092-48d0-8c45-966e2e0503c0 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:55.296638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605ab189-a092-48d0-8c45-966e2e0503c0 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:55.342753 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:55.349559 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28109632-54f0-470d-b703-170fedccb7a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:55.350205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28109632-54f0-470d-b703-170fedccb7a4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:55.350484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28109632-54f0-470d-b703-170fedccb7a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:55.350758 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28109632-54f0-470d-b703-170fedccb7a4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:55.355543 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28109632-54f0-470d-b703-170fedccb7a4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:55.357059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28109632-54f0-470d-b703-170fedccb7a4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:55.359097 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 71/140] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:04:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:55.364195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766b92b9-9976-4f11-ac90-1236a17955bd None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:55.387383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:55.494877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766b92b9-9976-4f11-ac90-1236a17955bd None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:55.541937 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314dd878-75de-46ab-8ae6-0c327aea78f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:55.542566 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314dd878-75de-46ab-8ae6-0c327aea78f8 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:55.542925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314dd878-75de-46ab-8ae6-0c327aea78f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:55.543202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314dd878-75de-46ab-8ae6-0c327aea78f8 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:55.543867 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314dd878-75de-46ab-8ae6-0c327aea78f8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:55.545335 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314dd878-75de-46ab-8ae6-0c327aea78f8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:55.583224 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 70/141] 10.222.0.15 () {60 vars in 1132 bytes} [Tue Sep 13 01:04:55 2022] POST /identity/v3/users => generated 283 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:04:56.076452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c8fbcc-e0aa-4604-805f-4e42ef40cbd1 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:56.076842 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c8fbcc-e0aa-4604-805f-4e42ef40cbd1 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:56.077048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c8fbcc-e0aa-4604-805f-4e42ef40cbd1 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:56.078542 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 72/142] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:56.085086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8a25e3-a26c-4546-8f98-04db9aa50534 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:56.085374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8a25e3-a26c-4546-8f98-04db9aa50534 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:56.085649 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8a25e3-a26c-4546-8f98-04db9aa50534 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:56.108333 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-1e8a25e3-a26c-4546-8f98-04db9aa50534 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:56.109539 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1e8a25e3-a26c-4546-8f98-04db9aa50534 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:56.195031 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:56.201448 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 71/143] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:56.449937 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7aa1b76-df4f-4dfa-916e-1385734e458c None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:56.450426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7aa1b76-df4f-4dfa-916e-1385734e458c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:56.450498 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7aa1b76-df4f-4dfa-916e-1385734e458c None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:56.451925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 73/144] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:56.457309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c6ea8f-614b-4d82-8756-dbfe4c91cf9e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:56.458115 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c6ea8f-614b-4d82-8756-dbfe4c91cf9e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:56.458587 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c6ea8f-614b-4d82-8756-dbfe4c91cf9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:56.482720 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-71c6ea8f-614b-4d82-8756-dbfe4c91cf9e None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:56.484256 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-71c6ea8f-614b-4d82-8756-dbfe4c91cf9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:56.576038 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 72/145] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:56.634739 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7aa1b76-df4f-4dfa-916e-1385734e458c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:56.748213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7aa1b76-df4f-4dfa-916e-1385734e458c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:56.791810 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb5277f-2d5c-412a-940f-988976a11944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:56.792462 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb5277f-2d5c-412a-940f-988976a11944 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:56.792747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb5277f-2d5c-412a-940f-988976a11944 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:56.793025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb5277f-2d5c-412a-940f-988976a11944 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:56.807081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb5277f-2d5c-412a-940f-988976a11944 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:04:56.811920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-6fb5277f-2d5c-412a-940f-988976a11944 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 01:04:56.811920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:04:56.811920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:04:56.811920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:04:56.811920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:04:56.811920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:04:56.811920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:04:56.811920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:04:56.811920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:04:56.811920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:04:56.811920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:04:56.811920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:04:56.811920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:04:56.811920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:04:56.811920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:04:56.811920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:04:56.813688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:04:56.813688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:04:56.813688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:04:56.813688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:04:56.813688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:04:56.813688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:04:56.813688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:04:56.813688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:04:56.813688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:04:56.813688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:04:56.813688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:04:56.813688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:04:56.813688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:04:56.813688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:04:56.813688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:04:56.813688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:04:56.815683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:04:56.815683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:04:56.815683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:04:56.815683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:04:56.815683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:04:56.815683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:04:56.815683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:04:56.815683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:04:56.815683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:04:56.815683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:04:56.815683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 01:04:56.815683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:04:56.815683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 74/146] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:04:56 2022] GET /identity/v3/roles/reader => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:04:56.819564 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c6ea8f-614b-4d82-8756-dbfe4c91cf9e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:56.908831 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd867d62-854c-41ef-850b-69233fd3d259 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:56.908831 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd867d62-854c-41ef-850b-69233fd3d259 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:56.910828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd867d62-854c-41ef-850b-69233fd3d259 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:56.910828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 75/147] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:56.921989 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c5ece6-43e4-4caf-86a0-1b8bf8b7b2d9 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:56.922279 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c5ece6-43e4-4caf-86a0-1b8bf8b7b2d9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:56.922560 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c5ece6-43e4-4caf-86a0-1b8bf8b7b2d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:56.942968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c6ea8f-614b-4d82-8756-dbfe4c91cf9e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:56.959876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-85c5ece6-43e4-4caf-86a0-1b8bf8b7b2d9 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:56.961091 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-85c5ece6-43e4-4caf-86a0-1b8bf8b7b2d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:56.972737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:56.985739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1fcf90-fcd3-488a-89fa-c874ce12990f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:56.986601 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1fcf90-fcd3-488a-89fa-c874ce12990f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:56.987650 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1fcf90-fcd3-488a-89fa-c874ce12990f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:56.988180 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1fcf90-fcd3-488a-89fa-c874ce12990f None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:56.988851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1fcf90-fcd3-488a-89fa-c874ce12990f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:56.989152 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1fcf90-fcd3-488a-89fa-c874ce12990f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:56.990520 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1fcf90-fcd3-488a-89fa-c874ce12990f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:57.010137 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 73/148] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:04:56 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:57.012078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1fcf90-fcd3-488a-89fa-c874ce12990f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:57.096897 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:57.122602 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 76/149] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 01:04:57.123466 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c5ece6-43e4-4caf-86a0-1b8bf8b7b2d9 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:57.137687 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1fcf90-fcd3-488a-89fa-c874ce12990f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:57.178065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa367b9-11c1-4606-bf66-9e45a3e028eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:57.178984 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa367b9-11c1-4606-bf66-9e45a3e028eb None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:57.179414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa367b9-11c1-4606-bf66-9e45a3e028eb None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:57.179920 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa367b9-11c1-4606-bf66-9e45a3e028eb None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:57.194547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa367b9-11c1-4606-bf66-9e45a3e028eb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:04:57.201032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-8aa367b9-11c1-4606-bf66-9e45a3e028eb None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:04:57.201032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:04:57.201032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:04:57.201032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:04:57.201032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:04:57.201032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:04:57.201032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:04:57.201032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:04:57.201032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:04:57.201032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:04:57.201032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:04:57.201032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:04:57.201032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:04:57.201032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:04:57.201032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:04:57.201032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:04:57.202120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:04:57.202120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:04:57.202120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:04:57.202120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:04:57.202120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:04:57.202120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:04:57.202120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:04:57.202120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:04:57.202120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:04:57.202120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:04:57.202120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:04:57.202120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:04:57.202120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:04:57.202120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:04:57.202120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:04:57.202120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:04:57.207474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:04:57.207474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:04:57.207474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:04:57.207474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:04:57.207474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:04:57.207474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:04:57.207474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:04:57.207474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:04:57.207474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:04:57.207474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:04:57.207474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:04:57.207474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:04:57.207474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 74/150] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:04:57 2022] GET /identity/v3/roles/member => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:04:57.214564 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa367b9-11c1-4606-bf66-9e45a3e028eb None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:57.336345 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:57.341360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c5ece6-43e4-4caf-86a0-1b8bf8b7b2d9 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:57.407846 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c583b6-ea9a-4083-9523-19e841a4d6ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:57.408496 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c583b6-ea9a-4083-9523-19e841a4d6ee None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:57.408776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c583b6-ea9a-4083-9523-19e841a4d6ee None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:57.409051 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c583b6-ea9a-4083-9523-19e841a4d6ee None admin] PATH_INFO: `/v3/users/f7c6f07cdd5f4461ae915f6a9d72b7ca` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:57.410186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c583b6-ea9a-4083-9523-19e841a4d6ee None admin] RBAC: Authorizing `identity:get_user(user_id=f7c6f07cdd5f4461ae915f6a9d72b7ca)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:57.417307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c583b6-ea9a-4083-9523-19e841a4d6ee None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:57.417307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 77/151] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:04:57 2022] GET /identity/v3/users/f7c6f07cdd5f4461ae915f6a9d72b7ca => generated 289 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:57.423736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c583b6-ea9a-4083-9523-19e841a4d6ee None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:57.461114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa367b9-11c1-4606-bf66-9e45a3e028eb None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:57.510557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f9f609-11b4-49e8-b829-811a00c7793e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:57.511755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f9f609-11b4-49e8-b829-811a00c7793e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:57.512258 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f9f609-11b4-49e8-b829-811a00c7793e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:57.512722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f9f609-11b4-49e8-b829-811a00c7793e None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:57.513632 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f9f609-11b4-49e8-b829-811a00c7793e None admin] RBAC: Adding query filter params (name=member) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:57.514138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f9f609-11b4-49e8-b829-811a00c7793e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:57.515843 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f9f609-11b4-49e8-b829-811a00c7793e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:57.525925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 75/152] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:04:57 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:57.531799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:57.533910 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f9f609-11b4-49e8-b829-811a00c7793e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:57.612042 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c583b6-ea9a-4083-9523-19e841a4d6ee None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:57.648559 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f9f609-11b4-49e8-b829-811a00c7793e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:57.663497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:57.688246 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be55af8-31fd-42ba-a7dc-c3b4090771e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:57.688911 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be55af8-31fd-42ba-a7dc-c3b4090771e8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:57.689171 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be55af8-31fd-42ba-a7dc-c3b4090771e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:57.689449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be55af8-31fd-42ba-a7dc-c3b4090771e8 None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:57.690588 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be55af8-31fd-42ba-a7dc-c3b4090771e8 None admin] RBAC: Authorizing `identity:get_project(project_id=4426c2e7285b4cbc8331aa86690ef664)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:57.691222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2024d9-a3b4-4bae-9673-bc88eb85aa2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:57.692080 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2024d9-a3b4-4bae-9673-bc88eb85aa2c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:57.692196 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be55af8-31fd-42ba-a7dc-c3b4090771e8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:57.692671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2024d9-a3b4-4bae-9673-bc88eb85aa2c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:57.693140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2024d9-a3b4-4bae-9673-bc88eb85aa2c None admin] PATH_INFO: `/v3/users/1e80dcbdbef640078cfec6d97eabca95` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:57.693735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 78/153] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:04:57 2022] GET /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664 => generated 298 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:57.694516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be55af8-31fd-42ba-a7dc-c3b4090771e8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:57.695279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2024d9-a3b4-4bae-9673-bc88eb85aa2c None admin] RBAC: Authorizing `identity:get_user(user_id=1e80dcbdbef640078cfec6d97eabca95)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:57.696976 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2024d9-a3b4-4bae-9673-bc88eb85aa2c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:57.698949 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 76/154] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:04:57 2022] GET /identity/v3/users/1e80dcbdbef640078cfec6d97eabca95 => generated 275 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:57.703167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2024d9-a3b4-4bae-9673-bc88eb85aa2c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:57.819540 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2024d9-a3b4-4bae-9673-bc88eb85aa2c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:57.858785 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9c443d-cd3e-4b1a-b052-f2b8ae6726c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:57.859785 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9c443d-cd3e-4b1a-b052-f2b8ae6726c5 None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:57.860244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9c443d-cd3e-4b1a-b052-f2b8ae6726c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:57.860693 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9c443d-cd3e-4b1a-b052-f2b8ae6726c5 None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664/users/f7c6f07cdd5f4461ae915f6a9d72b7ca/roles/6f98018cf28149aa84201e6dc4a43c2d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:57.861964 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be55af8-31fd-42ba-a7dc-c3b4090771e8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:57.863096 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9c443d-cd3e-4b1a-b052-f2b8ae6726c5 None admin] RBAC: Authorizing `identity:create_grant(project_id=4426c2e7285b4cbc8331aa86690ef664, user_id=f7c6f07cdd5f4461ae915f6a9d72b7ca, role_id=6f98018cf28149aa84201e6dc4a43c2d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:57.864936 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9c443d-cd3e-4b1a-b052-f2b8ae6726c5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:57.885988 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 77/155] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:04:57 2022] PUT /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664/users/f7c6f07cdd5f4461ae915f6a9d72b7ca/roles/6f98018cf28149aa84201e6dc4a43c2d => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:04:57.891363 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:57.891997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9c443d-cd3e-4b1a-b052-f2b8ae6726c5 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:57.906888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6540770c-d89b-4f6f-838a-90b7ca299349 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:57.907560 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6540770c-d89b-4f6f-838a-90b7ca299349 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:57.908152 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6540770c-d89b-4f6f-838a-90b7ca299349 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:57.908152 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6540770c-d89b-4f6f-838a-90b7ca299349 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:57.925556 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6540770c-d89b-4f6f-838a-90b7ca299349 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:04:57.931417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-6540770c-d89b-4f6f-838a-90b7ca299349 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:04:57.931417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:04:57.931417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:04:57.931417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:04:57.931417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:04:57.931417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:04:57.931417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:04:57.931417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:04:57.931417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:04:57.931417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:04:57.931417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:04:57.931417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:04:57.931417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:04:57.931417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:04:57.931417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:04:57.931417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:04:57.932516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:04:57.932516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:04:57.932516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:04:57.932516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:04:57.932516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:04:57.932516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:04:57.932516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:04:57.932516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:04:57.932516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:04:57.932516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:04:57.932516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:04:57.932516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:04:57.932516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:04:57.932516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:04:57.932516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:04:57.932516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:04:57.933995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:04:57.933995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:04:57.933995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:04:57.933995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:04:57.933995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:04:57.933995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:04:57.933995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:04:57.933995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:04:57.933995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:04:57.933995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:04:57.933995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:04:57.933995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:04:57.933995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 79/156] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:04:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:04:57.938967 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6540770c-d89b-4f6f-838a-90b7ca299349 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:57.966808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:58.075961 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9c443d-cd3e-4b1a-b052-f2b8ae6726c5 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:58.149839 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6540770c-d89b-4f6f-838a-90b7ca299349 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:58.159061 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d560b77-d3d4-4d11-b649-21a7a0f0f521 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:58.160298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d560b77-d3d4-4d11-b649-21a7a0f0f521 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:58.160783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d560b77-d3d4-4d11-b649-21a7a0f0f521 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:58.161247 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d560b77-d3d4-4d11-b649-21a7a0f0f521 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:58.210243 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b395a7-5c81-4ab7-b209-7db24c053faa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:58.210945 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b395a7-5c81-4ab7-b209-7db24c053faa None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:58.211230 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b395a7-5c81-4ab7-b209-7db24c053faa None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:58.211566 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b395a7-5c81-4ab7-b209-7db24c053faa None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:58.212854 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b395a7-5c81-4ab7-b209-7db24c053faa None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:58.214894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b395a7-5c81-4ab7-b209-7db24c053faa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:58.214894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b395a7-5c81-4ab7-b209-7db24c053faa None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:58.229085 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 80/157] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:04:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:58.234956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b395a7-5c81-4ab7-b209-7db24c053faa None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:58.311662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d560b77-d3d4-4d11-b649-21a7a0f0f521 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:58.313800 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d560b77-d3d4-4d11-b649-21a7a0f0f521 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:58.322657 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:58.376867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b395a7-5c81-4ab7-b209-7db24c053faa None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:58.415882 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:58.455848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6a325c-f33c-4582-891e-f9282c1613a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:58.456560 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6a325c-f33c-4582-891e-f9282c1613a3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:58.456847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6a325c-f33c-4582-891e-f9282c1613a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:58.457122 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6a325c-f33c-4582-891e-f9282c1613a3 None admin] PATH_INFO: `/v3/users/22ddb5151c0f4696a4f99a9acc8804ea` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:58.459533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6a325c-f33c-4582-891e-f9282c1613a3 None admin] RBAC: Authorizing `identity:get_user(user_id=22ddb5151c0f4696a4f99a9acc8804ea)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:58.460993 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6a325c-f33c-4582-891e-f9282c1613a3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:58.463200 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 81/158] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:04:58 2022] GET /identity/v3/users/22ddb5151c0f4696a4f99a9acc8804ea => generated 247 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:58.471445 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6a325c-f33c-4582-891e-f9282c1613a3 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:58.503489 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 78/159] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:04:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 617 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:04:58.509526 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d560b77-d3d4-4d11-b649-21a7a0f0f521 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:58.619271 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6a325c-f33c-4582-891e-f9282c1613a3 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:58.663377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:58.700132 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafc62e0-47eb-41e0-a899-405476beb8b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:58.700742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafc62e0-47eb-41e0-a899-405476beb8b7 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:58.701016 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafc62e0-47eb-41e0-a899-405476beb8b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:58.701281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafc62e0-47eb-41e0-a899-405476beb8b7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:58.710339 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d560b77-d3d4-4d11-b649-21a7a0f0f521 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:58.813053 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a99c54d-8116-438e-a09e-25bbffcdff00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:58.814098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a99c54d-8116-438e-a09e-25bbffcdff00 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:58.814640 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a99c54d-8116-438e-a09e-25bbffcdff00 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:58.815156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a99c54d-8116-438e-a09e-25bbffcdff00 None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:58.816605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a99c54d-8116-438e-a09e-25bbffcdff00 None admin] RBAC: Authorizing `identity:get_project(project_id=4426c2e7285b4cbc8331aa86690ef664)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:58.822353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a99c54d-8116-438e-a09e-25bbffcdff00 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:58.824666 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 79/160] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:04:58 2022] GET /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664 => generated 298 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:58.836337 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a99c54d-8116-438e-a09e-25bbffcdff00 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:58.857371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:58.877173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafc62e0-47eb-41e0-a899-405476beb8b7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:58.878686 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafc62e0-47eb-41e0-a899-405476beb8b7 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:59.051017 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a99c54d-8116-438e-a09e-25bbffcdff00 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:59.113665 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 82/161] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:04:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 643 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:04:59.115253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6739973-acd6-400a-b66f-c989ac0dc0c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:59.115687 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163d4e73-6889-4324-ab96-c0a236b5da1b None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:59.115859 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6739973-acd6-400a-b66f-c989ac0dc0c2 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:59.116144 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6739973-acd6-400a-b66f-c989ac0dc0c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:59.116417 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6739973-acd6-400a-b66f-c989ac0dc0c2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:59.116767 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163d4e73-6889-4324-ab96-c0a236b5da1b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:59.117151 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6739973-acd6-400a-b66f-c989ac0dc0c2 None admin] RBAC: Adding query filter params (role.id=6f71034643cb4e8caee299f5d0bb863f, scope.project.id=4426c2e7285b4cbc8331aa86690ef664, user.id=1e80dcbdbef640078cfec6d97eabca95) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:59.117434 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163d4e73-6889-4324-ab96-c0a236b5da1b None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:59.117538 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6739973-acd6-400a-b66f-c989ac0dc0c2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:59.119557 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 83/162] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:59 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:59.120950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8068523-6d0c-4880-89b2-1b901ca62c49 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:59.121438 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8068523-6d0c-4880-89b2-1b901ca62c49 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:59.121894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8068523-6d0c-4880-89b2-1b901ca62c49 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:59.123285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 84/163] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:04:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:04:59.123823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6739973-acd6-400a-b66f-c989ac0dc0c2 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:59.124397 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8068523-6d0c-4880-89b2-1b901ca62c49 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:59.136811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 80/164] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:04:58 2022] GET /identity/v3/role_assignments?user.id=1e80dcbdbef640078cfec6d97eabca95&scope.project.id=4426c2e7285b4cbc8331aa86690ef664&role.id=6f71034643cb4e8caee299f5d0bb863f => generated 608 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:04:59.138052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c62083f-afb6-48bd-9633-324469ea3c15 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:59.138335 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c62083f-afb6-48bd-9633-324469ea3c15 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:59.138610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c62083f-afb6-48bd-9633-324469ea3c15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:59.169132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-2c62083f-afb6-48bd-9633-324469ea3c15 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:59.170397 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2c62083f-afb6-48bd-9633-324469ea3c15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:59.187218 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:59.277403 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 81/165] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:59.278768 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b02ea9-adc1-427a-9888-86d048a2c019 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:59.279084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b02ea9-adc1-427a-9888-86d048a2c019 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:59.279362 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b02ea9-adc1-427a-9888-86d048a2c019 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:59.295248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:59.298603 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-57b02ea9-adc1-427a-9888-86d048a2c019 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:04:59.300639 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-57b02ea9-adc1-427a-9888-86d048a2c019 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:04:59.389835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 82/166] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:04:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:04:59.452582 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8068523-6d0c-4880-89b2-1b901ca62c49 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:59.497510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e7e726-05ae-4557-b911-e96a82895415 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:59.498156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e7e726-05ae-4557-b911-e96a82895415 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:59.498427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e7e726-05ae-4557-b911-e96a82895415 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:59.498694 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e7e726-05ae-4557-b911-e96a82895415 None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:59.503525 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e7e726-05ae-4557-b911-e96a82895415 None admin] RBAC: Authorizing `identity:get_project(project_id=4426c2e7285b4cbc8331aa86690ef664)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:59.504966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e7e726-05ae-4557-b911-e96a82895415 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:59.506609 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 85/167] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:04:59 2022] GET /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664 => generated 298 bytes in 383 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:59.513355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b02ea9-adc1-427a-9888-86d048a2c019 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:59.670731 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b02ea9-adc1-427a-9888-86d048a2c019 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:04:59.723967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:59.732481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff91a8ae-60cb-4ab9-888f-c3c80874bffb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:04:59.733103 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff91a8ae-60cb-4ab9-888f-c3c80874bffb None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:04:59.733382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff91a8ae-60cb-4ab9-888f-c3c80874bffb None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:04:59.733654 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff91a8ae-60cb-4ab9-888f-c3c80874bffb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:04:59.734386 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff91a8ae-60cb-4ab9-888f-c3c80874bffb None admin] RBAC: Adding query filter params (role.id=66a36c858aab40688080eed6ceace3a0, scope.project.id=4426c2e7285b4cbc8331aa86690ef664, user.id=22ddb5151c0f4696a4f99a9acc8804ea) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:04:59.734691 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff91a8ae-60cb-4ab9-888f-c3c80874bffb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:04:59.736209 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff91a8ae-60cb-4ab9-888f-c3c80874bffb None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:04:59.743576 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 83/168] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:04:59 2022] GET /identity/v3/role_assignments?user.id=22ddb5151c0f4696a4f99a9acc8804ea&scope.project.id=4426c2e7285b4cbc8331aa86690ef664&role.id=66a36c858aab40688080eed6ceace3a0 => generated 608 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:04:59.876604 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e7e726-05ae-4557-b911-e96a82895415 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:59.969965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:04:59.973898 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff91a8ae-60cb-4ab9-888f-c3c80874bffb None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:04:59.994350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e7e726-05ae-4557-b911-e96a82895415 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:00.057523 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce46703-90e5-4f8b-8257-3811d1e11b94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:00.058179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce46703-90e5-4f8b-8257-3811d1e11b94 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:00.058436 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce46703-90e5-4f8b-8257-3811d1e11b94 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:00.058702 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce46703-90e5-4f8b-8257-3811d1e11b94 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:00.091882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce46703-90e5-4f8b-8257-3811d1e11b94 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:00.096749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-fce46703-90e5-4f8b-8257-3811d1e11b94 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:00.096749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:00.096749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:00.096749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:00.096749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:00.096749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:00.096749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:00.096749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:00.096749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:00.096749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:00.096749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:00.096749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:00.096749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:00.096749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:00.096749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:00.096749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:00.098827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:00.098827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:00.098827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:00.098827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:00.098827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:00.098827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:00.098827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:00.098827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:00.098827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:00.098827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:00.098827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:00.098827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:00.098827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:00.098827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:00.098827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:00.098827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:00.100848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:00.100848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:00.100848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:00.100848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:00.100848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:00.100848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:00.100848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:00.100848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:00.100848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:00.100848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:00.100848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:00.100848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:00.100848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 86/169] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:04:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:00.108084 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce46703-90e5-4f8b-8257-3811d1e11b94 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:00.162903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff91a8ae-60cb-4ab9-888f-c3c80874bffb None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:00.211030 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3549f76f-7d1e-4371-b46f-f5e6dcda7299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:00.212146 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3549f76f-7d1e-4371-b46f-f5e6dcda7299 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:00.212653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3549f76f-7d1e-4371-b46f-f5e6dcda7299 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:00.213144 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3549f76f-7d1e-4371-b46f-f5e6dcda7299 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:00.229472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3549f76f-7d1e-4371-b46f-f5e6dcda7299 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:00.235171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-3549f76f-7d1e-4371-b46f-f5e6dcda7299 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:00.235171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:00.235171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:00.235171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:00.235171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:00.235171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:00.235171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:00.235171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:00.235171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:00.235171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:00.235171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:00.235171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:00.235171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:00.235171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:00.235171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:00.235171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:00.236009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:00.236009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:00.236009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:00.236009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:00.236009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:00.236009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:00.236009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:00.236009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:00.236009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:00.236009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:00.236009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:00.236009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:00.236009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:00.236009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:00.236009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:00.236009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:00.236782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:00.236782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:00.236782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:00.236782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:00.236782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:00.236782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:00.236782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:00.236782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:00.236782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:00.236782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:00.236782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:00.236782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:00.243215 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 84/170] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:04:59 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:00.249153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3549f76f-7d1e-4371-b46f-f5e6dcda7299 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:00.328115 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:00.359964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3549f76f-7d1e-4371-b46f-f5e6dcda7299 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:00.396661 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ad0e4a-6a17-45fc-9dbf-3306785b0004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:00.397657 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ad0e4a-6a17-45fc-9dbf-3306785b0004 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:00.398156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ad0e4a-6a17-45fc-9dbf-3306785b0004 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:00.398634 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ad0e4a-6a17-45fc-9dbf-3306785b0004 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:00.399601 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ad0e4a-6a17-45fc-9dbf-3306785b0004 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:00.400123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ad0e4a-6a17-45fc-9dbf-3306785b0004 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:00.401727 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ad0e4a-6a17-45fc-9dbf-3306785b0004 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:00.405078 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce46703-90e5-4f8b-8257-3811d1e11b94 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:00.411427 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 85/171] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:00 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:00.417832 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ad0e4a-6a17-45fc-9dbf-3306785b0004 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:00.508909 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72877b53-e0c5-4080-890f-92fa766190ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:00.509454 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72877b53-e0c5-4080-890f-92fa766190ee None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:00.509732 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72877b53-e0c5-4080-890f-92fa766190ee None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:00.510002 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72877b53-e0c5-4080-890f-92fa766190ee None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:00.510696 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72877b53-e0c5-4080-890f-92fa766190ee None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:00.511009 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72877b53-e0c5-4080-890f-92fa766190ee None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:00.512403 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72877b53-e0c5-4080-890f-92fa766190ee None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:00.538302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ad0e4a-6a17-45fc-9dbf-3306785b0004 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:00.541090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 87/172] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 433 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:00.556468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72877b53-e0c5-4080-890f-92fa766190ee None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:00.566141 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:00.570293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:00.611584 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4565a0-256f-426f-8ba4-99179a336ff3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:00.612242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4565a0-256f-426f-8ba4-99179a336ff3 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:00.612525 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4565a0-256f-426f-8ba4-99179a336ff3 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:00.612803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4565a0-256f-426f-8ba4-99179a336ff3 None admin] PATH_INFO: `/v3/users/1e80dcbdbef640078cfec6d97eabca95` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:00.614058 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4565a0-256f-426f-8ba4-99179a336ff3 None admin] RBAC: Authorizing `identity:get_user(user_id=1e80dcbdbef640078cfec6d97eabca95)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:00.615530 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4565a0-256f-426f-8ba4-99179a336ff3 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:00.617117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 86/173] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:00 2022] GET /identity/v3/users/1e80dcbdbef640078cfec6d97eabca95 => generated 275 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:00.623242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4565a0-256f-426f-8ba4-99179a336ff3 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:00.714142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72877b53-e0c5-4080-890f-92fa766190ee None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:00.773520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b665dd-8819-4bab-808a-46e799cbe336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:00.774089 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b665dd-8819-4bab-808a-46e799cbe336 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:00.774923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b665dd-8819-4bab-808a-46e799cbe336 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:00.775241 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b665dd-8819-4bab-808a-46e799cbe336 None admin] PATH_INFO: `/v3/users/1e80dcbdbef640078cfec6d97eabca95` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:00.776631 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b665dd-8819-4bab-808a-46e799cbe336 None admin] RBAC: Authorizing `identity:get_user(user_id=1e80dcbdbef640078cfec6d97eabca95)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:00.782948 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b665dd-8819-4bab-808a-46e799cbe336 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:00.782948 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 88/174] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:00 2022] GET /identity/v3/users/1e80dcbdbef640078cfec6d97eabca95 => generated 275 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:00.782948 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e2372a-6b41-4d0e-afaf-60a805d03d8b None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:00.782948 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e2372a-6b41-4d0e-afaf-60a805d03d8b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:00.782948 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e2372a-6b41-4d0e-afaf-60a805d03d8b None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:00.784763 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 89/175] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:00.786704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e2372a-6b41-4d0e-afaf-60a805d03d8b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:00.814779 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4565a0-256f-426f-8ba4-99179a336ff3 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:00.854653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15a7e23-1860-4526-a828-577644385828 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:00.855309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15a7e23-1860-4526-a828-577644385828 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:00.855588 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15a7e23-1860-4526-a828-577644385828 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:00.855862 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15a7e23-1860-4526-a828-577644385828 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:00.954014 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:00.971212 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:00.995984 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15a7e23-1860-4526-a828-577644385828 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:00.997470 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15a7e23-1860-4526-a828-577644385828 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:01.009890 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e2372a-6b41-4d0e-afaf-60a805d03d8b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:01.091115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1768c7-e386-42d2-92e6-c2452de68f57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:01.091647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1768c7-e386-42d2-92e6-c2452de68f57 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:01.091932 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1768c7-e386-42d2-92e6-c2452de68f57 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:01.092254 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1768c7-e386-42d2-92e6-c2452de68f57 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:01.182968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 87/176] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 560 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 13 01:05:01.184311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedace99-7e1e-4216-97bb-9bcc171d89d0 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:01.184602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedace99-7e1e-4216-97bb-9bcc171d89d0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:01.184881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedace99-7e1e-4216-97bb-9bcc171d89d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:01.191920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1768c7-e386-42d2-92e6-c2452de68f57 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:01.193581 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1768c7-e386-42d2-92e6-c2452de68f57 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:01.204013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-cedace99-7e1e-4216-97bb-9bcc171d89d0 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:01.205630 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-cedace99-7e1e-4216-97bb-9bcc171d89d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:01.275799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:01.284794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 88/177] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:01.286113 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bd078e-d1ad-4919-b9aa-766fd6ca2bc9 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:01.286390 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bd078e-d1ad-4919-b9aa-766fd6ca2bc9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:01.286811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bd078e-d1ad-4919-b9aa-766fd6ca2bc9 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:01.288097 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 89/178] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:01.288611 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bd078e-d1ad-4919-b9aa-766fd6ca2bc9 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:01.310875 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:01.311315 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 90/179] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 525 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:01.312840 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fab8e25-ab2e-4504-9691-c0b9dc6ad776 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:01.313134 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fab8e25-ab2e-4504-9691-c0b9dc6ad776 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:01.314261 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fab8e25-ab2e-4504-9691-c0b9dc6ad776 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:01.314995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 91/180] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:01.320823 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e5ea76-dd9a-488b-ad42-43198368b6d4 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:01.321113 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e5ea76-dd9a-488b-ad42-43198368b6d4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:01.321440 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e5ea76-dd9a-488b-ad42-43198368b6d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:01.342152 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-06e5ea76-dd9a-488b-ad42-43198368b6d4 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:01.343755 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-06e5ea76-dd9a-488b-ad42-43198368b6d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:01.423734 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 92/181] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:01.425058 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e5ea76-dd9a-488b-ad42-43198368b6d4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:01.433539 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bd078e-d1ad-4919-b9aa-766fd6ca2bc9 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:01.505769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe86202e-57b1-4135-a3fc-b7f9df9d73fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:01.506778 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe86202e-57b1-4135-a3fc-b7f9df9d73fb None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:01.507277 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe86202e-57b1-4135-a3fc-b7f9df9d73fb None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:01.507745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe86202e-57b1-4135-a3fc-b7f9df9d73fb None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:01.509266 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe86202e-57b1-4135-a3fc-b7f9df9d73fb None admin] RBAC: Authorizing `identity:get_project(project_id=4426c2e7285b4cbc8331aa86690ef664)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:01.512245 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe86202e-57b1-4135-a3fc-b7f9df9d73fb None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:01.519759 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 90/182] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:01 2022] GET /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664 => generated 298 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:01.522617 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a8c3f3-5cc5-4157-89ea-b3f1ed6d1ccb None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:01.523334 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a8c3f3-5cc5-4157-89ea-b3f1ed6d1ccb None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:01.525102 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a8c3f3-5cc5-4157-89ea-b3f1ed6d1ccb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:01.553776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-39a8c3f3-5cc5-4157-89ea-b3f1ed6d1ccb None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:01.556363 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-39a8c3f3-5cc5-4157-89ea-b3f1ed6d1ccb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:01.588868 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e5ea76-dd9a-488b-ad42-43198368b6d4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:01.591295 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:01.624992 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d92ea9-0ad1-459f-b87e-2757ad019469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:01.625800 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d92ea9-0ad1-459f-b87e-2757ad019469 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:01.625800 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d92ea9-0ad1-459f-b87e-2757ad019469 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:01.626070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d92ea9-0ad1-459f-b87e-2757ad019469 None admin] PATH_INFO: `/v3/projects/bf2e886952254e57a265bbc1fb8be1a1` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:01.627213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d92ea9-0ad1-459f-b87e-2757ad019469 None admin] RBAC: Authorizing `identity:get_project(project_id=bf2e886952254e57a265bbc1fb8be1a1)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:01.628635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d92ea9-0ad1-459f-b87e-2757ad019469 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:01.630514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 93/183] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:01 2022] GET /identity/v3/projects/bf2e886952254e57a265bbc1fb8be1a1 => generated 312 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:01.631019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d92ea9-0ad1-459f-b87e-2757ad019469 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:01.635801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 91/184] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:01.637050 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4f5e4a-6724-4609-83be-9eb321a769c5 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:01.637352 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4f5e4a-6724-4609-83be-9eb321a769c5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:01.637628 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4f5e4a-6724-4609-83be-9eb321a769c5 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:01.639421 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 92/185] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:01.644065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4f5e4a-6724-4609-83be-9eb321a769c5 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:01.660714 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:01.753421 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4f5e4a-6724-4609-83be-9eb321a769c5 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:01.756306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d92ea9-0ad1-459f-b87e-2757ad019469 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:01.790254 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6889335-70b2-43df-9094-4711e04be635 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:01.790888 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6889335-70b2-43df-9094-4711e04be635 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:01.791169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6889335-70b2-43df-9094-4711e04be635 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:01.791449 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6889335-70b2-43df-9094-4711e04be635 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:01.791975 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2904e98-d140-44df-bd84-dfc870c3b3bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:01.792578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2904e98-d140-44df-bd84-dfc870c3b3bb None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:01.792854 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2904e98-d140-44df-bd84-dfc870c3b3bb None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:01.793131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2904e98-d140-44df-bd84-dfc870c3b3bb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:01.793845 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2904e98-d140-44df-bd84-dfc870c3b3bb None admin] RBAC: Adding query filter params (role.id=cd616e816a394068867091116a3f1ce0, scope.project.id=4426c2e7285b4cbc8331aa86690ef664, user.id=1e80dcbdbef640078cfec6d97eabca95) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:01.794153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2904e98-d140-44df-bd84-dfc870c3b3bb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:01.795609 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2904e98-d140-44df-bd84-dfc870c3b3bb None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:01.802318 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6889335-70b2-43df-9094-4711e04be635 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:01.803589 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 94/186] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:05:01 2022] GET /identity/v3/role_assignments?user.id=1e80dcbdbef640078cfec6d97eabca95&scope.project.id=4426c2e7285b4cbc8331aa86690ef664&role.id=cd616e816a394068867091116a3f1ce0 => generated 608 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:01.807140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-b6889335-70b2-43df-9094-4711e04be635 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:05:01.807140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:01.807140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:01.807140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:01.807140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:01.807140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:01.807140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:01.807140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:01.807140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:01.807140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:01.807140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:01.807140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:01.807140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:01.807140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:01.807140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:01.807140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:01.808045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:01.808045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:01.808045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:01.808045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:01.808045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:01.808045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:01.808045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:01.808045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:01.808045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:01.808045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:01.808045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:01.808045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:01.808045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:01.808045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:01.808045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:01.808045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:01.808826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:01.808826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:01.808826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:01.808826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:01.808826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:01.808826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:01.808826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:01.808826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:01.808826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:01.808826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:01.808826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:05:01.808826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:01.810213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2904e98-d140-44df-bd84-dfc870c3b3bb None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:01.811468 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 93/187] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:05:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:01.814479 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea8e6a3-3a4a-4ab6-b59f-0ae2430a0515 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:01.814770 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea8e6a3-3a4a-4ab6-b59f-0ae2430a0515 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:01.815074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea8e6a3-3a4a-4ab6-b59f-0ae2430a0515 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:01.837135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4ea8e6a3-3a4a-4ab6-b59f-0ae2430a0515 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:01.839550 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4ea8e6a3-3a4a-4ab6-b59f-0ae2430a0515 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:01.865463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:01.921515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 94/188] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:01.923781 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea8e6a3-3a4a-4ab6-b59f-0ae2430a0515 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:01.928344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:01.976181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2904e98-d140-44df-bd84-dfc870c3b3bb None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:02.021393 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a652fb4-8197-4c10-afbb-4dab4e4a3187 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:02.022038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a652fb4-8197-4c10-afbb-4dab4e4a3187 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:02.022342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a652fb4-8197-4c10-afbb-4dab4e4a3187 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:02.022596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a652fb4-8197-4c10-afbb-4dab4e4a3187 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:02.023389 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a652fb4-8197-4c10-afbb-4dab4e4a3187 None admin] RBAC: Adding query filter params (role.id=6f71034643cb4e8caee299f5d0bb863f, scope.project.id=bf2e886952254e57a265bbc1fb8be1a1, user.id=1e80dcbdbef640078cfec6d97eabca95) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:02.023674 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a652fb4-8197-4c10-afbb-4dab4e4a3187 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:02.025731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a652fb4-8197-4c10-afbb-4dab4e4a3187 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:02.032489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 95/189] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:05:01 2022] GET /identity/v3/role_assignments?user.id=1e80dcbdbef640078cfec6d97eabca95&scope.project.id=bf2e886952254e57a265bbc1fb8be1a1&role.id=6f71034643cb4e8caee299f5d0bb863f => generated 608 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:02.033122 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a652fb4-8197-4c10-afbb-4dab4e4a3187 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:02.049235 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea8e6a3-3a4a-4ab6-b59f-0ae2430a0515 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:02.101890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206e7483-3b99-479a-8da5-876deb224332 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:02.102893 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206e7483-3b99-479a-8da5-876deb224332 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:02.103608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206e7483-3b99-479a-8da5-876deb224332 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:02.104089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206e7483-3b99-479a-8da5-876deb224332 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:02.122424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206e7483-3b99-479a-8da5-876deb224332 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:02.128943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-206e7483-3b99-479a-8da5-876deb224332 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:02.128943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:02.128943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:02.128943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:02.128943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:02.128943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:02.128943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:02.128943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:02.128943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:02.128943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:02.128943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:02.128943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:02.128943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:02.128943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:02.128943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:02.128943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:02.130095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:02.130095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:02.130095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:02.130095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:02.130095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:02.130095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:02.130095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:02.130095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:02.130095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:02.130095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:02.130095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:02.130095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:02.130095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:02.130095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:02.130095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:02.130095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:02.131078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:02.131078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:02.131078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:02.131078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:02.131078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:02.131078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:02.131078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:02.131078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:02.131078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:02.131078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:02.131078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:02.131078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:02.134194 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 95/190] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:05:01 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:02.135676 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206e7483-3b99-479a-8da5-876deb224332 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:02.164599 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:02.171037 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a652fb4-8197-4c10-afbb-4dab4e4a3187 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:02.222125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4458f605-75b9-4ccc-ba0d-a8efe2a927ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:02.222833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4458f605-75b9-4ccc-ba0d-a8efe2a927ab None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:02.223797 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:02.223916 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4458f605-75b9-4ccc-ba0d-a8efe2a927ab None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:02.224744 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4458f605-75b9-4ccc-ba0d-a8efe2a927ab None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:02.225458 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4458f605-75b9-4ccc-ba0d-a8efe2a927ab None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:02.225927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4458f605-75b9-4ccc-ba0d-a8efe2a927ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:02.227618 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4458f605-75b9-4ccc-ba0d-a8efe2a927ab None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:02.244902 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 96/191] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:02.244902 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4458f605-75b9-4ccc-ba0d-a8efe2a927ab None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:02.282799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206e7483-3b99-479a-8da5-876deb224332 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:02.319857 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bc8adf-6f22-4985-983b-9cbf493784f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:02.320754 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bc8adf-6f22-4985-983b-9cbf493784f6 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:02.321220 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bc8adf-6f22-4985-983b-9cbf493784f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:02.322048 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bc8adf-6f22-4985-983b-9cbf493784f6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:02.323231 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bc8adf-6f22-4985-983b-9cbf493784f6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:02.324686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bc8adf-6f22-4985-983b-9cbf493784f6 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:02.326458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 96/192] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:02 2022] GET /identity/v3/domains/default => generated 199 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:02.331183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bc8adf-6f22-4985-983b-9cbf493784f6 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:02.357756 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4458f605-75b9-4ccc-ba0d-a8efe2a927ab None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:02.390483 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4127679f-c280-4973-bdd1-acde73112b91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:02.391137 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4127679f-c280-4973-bdd1-acde73112b91 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:02.391422 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4127679f-c280-4973-bdd1-acde73112b91 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:02.391693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4127679f-c280-4973-bdd1-acde73112b91 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:02.392350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4127679f-c280-4973-bdd1-acde73112b91 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:02.392657 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4127679f-c280-4973-bdd1-acde73112b91 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:02.394022 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4127679f-c280-4973-bdd1-acde73112b91 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:02.404366 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 97/193] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:02 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:02.404366 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4127679f-c280-4973-bdd1-acde73112b91 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:02.428727 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bc8adf-6f22-4985-983b-9cbf493784f6 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:02.431520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:02.457677 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:02.463082 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8594c565-e412-46c8-b21c-d1464cd5cfac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:02.463699 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8594c565-e412-46c8-b21c-d1464cd5cfac None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:02.463969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8594c565-e412-46c8-b21c-d1464cd5cfac None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:02.464235 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8594c565-e412-46c8-b21c-d1464cd5cfac None admin] PATH_INFO: `/v3/users/33fab7c55b3a46b5aa91e225ea277804` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:02.477269 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8594c565-e412-46c8-b21c-d1464cd5cfac None admin] RBAC: Authorizing `identity:get_user(user_id=33fab7c55b3a46b5aa91e225ea277804)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:02.479216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8594c565-e412-46c8-b21c-d1464cd5cfac None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:02.480802 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 97/194] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:02 2022] GET /identity/v3/users/33fab7c55b3a46b5aa91e225ea277804 => generated 283 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:02.481561 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8594c565-e412-46c8-b21c-d1464cd5cfac None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:02.511346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4127679f-c280-4973-bdd1-acde73112b91 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:02.545301 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48462a47-9478-4f4e-85a7-3d1c43071bc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:02.545924 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48462a47-9478-4f4e-85a7-3d1c43071bc2 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:02.546210 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48462a47-9478-4f4e-85a7-3d1c43071bc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:02.546481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48462a47-9478-4f4e-85a7-3d1c43071bc2 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:02.556856 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48462a47-9478-4f4e-85a7-3d1c43071bc2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:02.562119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-48462a47-9478-4f4e-85a7-3d1c43071bc2 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 01:05:02.562119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:02.562119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:02.562119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:02.562119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:02.562119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:02.562119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:02.562119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:02.562119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:02.562119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:02.562119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:02.562119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:02.562119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:02.562119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:02.562119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:02.562119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:02.563334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:02.563334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:02.563334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:02.563334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:02.563334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:02.563334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:02.563334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:02.563334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:02.563334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:02.563334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:02.563334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:02.563334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:02.563334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:02.563334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:02.563334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:02.563334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:02.564108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:02.564108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:02.564108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:02.564108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:02.564108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:02.564108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:02.564108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:02.564108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:02.564108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:02.564108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:02.564108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 01:05:02.564108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:02.564108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 98/195] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:02 2022] GET /identity/v3/roles/reader => generated 84 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:02.564746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48462a47-9478-4f4e-85a7-3d1c43071bc2 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:02.581089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8594c565-e412-46c8-b21c-d1464cd5cfac None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:02.617166 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13b4db8-ae46-4d92-97ee-098ff1318f05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:02.617878 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13b4db8-ae46-4d92-97ee-098ff1318f05 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:02.618183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13b4db8-ae46-4d92-97ee-098ff1318f05 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:02.618454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13b4db8-ae46-4d92-97ee-098ff1318f05 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:02.619164 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13b4db8-ae46-4d92-97ee-098ff1318f05 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:02.620603 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13b4db8-ae46-4d92-97ee-098ff1318f05 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:02.642383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 98/196] 10.222.0.15 () {60 vars in 1132 bytes} [Tue Sep 13 01:05:02 2022] POST /identity/v3/users => generated 297 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:05:02.643223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13b4db8-ae46-4d92-97ee-098ff1318f05 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:02.651573 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:02.671942 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48462a47-9478-4f4e-85a7-3d1c43071bc2 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:02.708018 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:02.712038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beed282c-53e9-422b-966f-7c3c16383029 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:02.712680 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beed282c-53e9-422b-966f-7c3c16383029 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:02.712960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beed282c-53e9-422b-966f-7c3c16383029 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:02.713229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beed282c-53e9-422b-966f-7c3c16383029 None admin] PATH_INFO: `/v3/users/33fab7c55b3a46b5aa91e225ea277804` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:02.714388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beed282c-53e9-422b-966f-7c3c16383029 None admin] RBAC: Authorizing `identity:get_user(user_id=33fab7c55b3a46b5aa91e225ea277804)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:02.716361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beed282c-53e9-422b-966f-7c3c16383029 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:02.717582 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 99/197] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:02 2022] GET /identity/v3/users/33fab7c55b3a46b5aa91e225ea277804 => generated 283 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:02.723538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beed282c-53e9-422b-966f-7c3c16383029 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:02.771253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13b4db8-ae46-4d92-97ee-098ff1318f05 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:02.811384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81aed8cf-b43e-4cbf-be1f-cd995eb2642d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:02.812033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81aed8cf-b43e-4cbf-be1f-cd995eb2642d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:02.812314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81aed8cf-b43e-4cbf-be1f-cd995eb2642d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:02.813147 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81aed8cf-b43e-4cbf-be1f-cd995eb2642d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:02.841593 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beed282c-53e9-422b-966f-7c3c16383029 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:02.880302 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f23b2d6-dbd9-4fd7-84a5-d5c131ab3f6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:02.880915 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f23b2d6-dbd9-4fd7-84a5-d5c131ab3f6f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:02.881194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f23b2d6-dbd9-4fd7-84a5-d5c131ab3f6f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:02.881462 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f23b2d6-dbd9-4fd7-84a5-d5c131ab3f6f None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:02.882122 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f23b2d6-dbd9-4fd7-84a5-d5c131ab3f6f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:02.882426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f23b2d6-dbd9-4fd7-84a5-d5c131ab3f6f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:02.884244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f23b2d6-dbd9-4fd7-84a5-d5c131ab3f6f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:02.886396 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81aed8cf-b43e-4cbf-be1f-cd995eb2642d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:02.887862 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81aed8cf-b43e-4cbf-be1f-cd995eb2642d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:02.892820 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 100/198] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:02 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:02.893306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f23b2d6-dbd9-4fd7-84a5-d5c131ab3f6f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:02.911368 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:02.946983 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:03.018027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f23b2d6-dbd9-4fd7-84a5-d5c131ab3f6f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:03.022910 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 99/199] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:03.022910 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81aed8cf-b43e-4cbf-be1f-cd995eb2642d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:03.068908 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8112a905-9442-494d-b06f-9ec777f0ae5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:03.070058 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8112a905-9442-494d-b06f-9ec777f0ae5b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:03.070342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8112a905-9442-494d-b06f-9ec777f0ae5b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:03.070617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8112a905-9442-494d-b06f-9ec777f0ae5b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:03.132897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81aed8cf-b43e-4cbf-be1f-cd995eb2642d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:03.163773 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8112a905-9442-494d-b06f-9ec777f0ae5b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:03.166581 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8112a905-9442-494d-b06f-9ec777f0ae5b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:03.169651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d046374-78fb-426c-bc3d-8aa7e6929f34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:03.170486 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d046374-78fb-426c-bc3d-8aa7e6929f34 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:03.170962 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d046374-78fb-426c-bc3d-8aa7e6929f34 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:03.171408 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d046374-78fb-426c-bc3d-8aa7e6929f34 None admin] PATH_INFO: `/v3/users/f7c6f07cdd5f4461ae915f6a9d72b7ca` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:03.172738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d046374-78fb-426c-bc3d-8aa7e6929f34 None admin] RBAC: Authorizing `identity:get_user(user_id=f7c6f07cdd5f4461ae915f6a9d72b7ca)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:03.174488 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d046374-78fb-426c-bc3d-8aa7e6929f34 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:03.176295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 100/200] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:03 2022] GET /identity/v3/users/f7c6f07cdd5f4461ae915f6a9d72b7ca => generated 289 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:03.177298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d046374-78fb-426c-bc3d-8aa7e6929f34 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:03.184313 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:03.196209 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:03.294235 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d046374-78fb-426c-bc3d-8aa7e6929f34 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:03.303009 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 101/201] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:03.304134 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8112a905-9442-494d-b06f-9ec777f0ae5b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:03.342800 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8954b26b-2248-42e0-9d63-73e5be6c287f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:03.342800 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8954b26b-2248-42e0-9d63-73e5be6c287f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:03.342800 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8954b26b-2248-42e0-9d63-73e5be6c287f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:03.342800 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8954b26b-2248-42e0-9d63-73e5be6c287f None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:03.343656 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8954b26b-2248-42e0-9d63-73e5be6c287f None admin] RBAC: Authorizing `identity:get_project(project_id=16a6348ce8d34348b34d853f76445b3a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:03.345275 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8954b26b-2248-42e0-9d63-73e5be6c287f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:03.352985 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 101/202] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:03 2022] GET /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a => generated 302 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:03.352985 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8954b26b-2248-42e0-9d63-73e5be6c287f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:03.418937 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8112a905-9442-494d-b06f-9ec777f0ae5b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:03.435215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:03.450500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:03.457304 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e957de5c-d735-425d-b84b-ef850e4321cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:03.458763 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e957de5c-d735-425d-b84b-ef850e4321cb None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:03.459100 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e957de5c-d735-425d-b84b-ef850e4321cb None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:03.459376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e957de5c-d735-425d-b84b-ef850e4321cb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:03.471426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8954b26b-2248-42e0-9d63-73e5be6c287f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:03.507167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a898b4d8-bb39-4348-8bcc-5cc56f0dbdf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:03.508251 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a898b4d8-bb39-4348-8bcc-5cc56f0dbdf8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:03.508823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a898b4d8-bb39-4348-8bcc-5cc56f0dbdf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:03.509286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a898b4d8-bb39-4348-8bcc-5cc56f0dbdf8 None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:03.510610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a898b4d8-bb39-4348-8bcc-5cc56f0dbdf8 None admin] RBAC: Authorizing `identity:get_project(project_id=16a6348ce8d34348b34d853f76445b3a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:03.512267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a898b4d8-bb39-4348-8bcc-5cc56f0dbdf8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:03.514120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 102/203] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:03 2022] GET /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a => generated 302 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:03.515242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a898b4d8-bb39-4348-8bcc-5cc56f0dbdf8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:03.537839 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e957de5c-d735-425d-b84b-ef850e4321cb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:03.539465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e957de5c-d735-425d-b84b-ef850e4321cb None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:03.621668 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a898b4d8-bb39-4348-8bcc-5cc56f0dbdf8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:03.652704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 102/204] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:03.653698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e957de5c-d735-425d-b84b-ef850e4321cb None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:03.658828 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d52927-01dd-42da-9f6f-501c00a4110c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:03.659775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d52927-01dd-42da-9f6f-501c00a4110c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:03.660270 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d52927-01dd-42da-9f6f-501c00a4110c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:03.660746 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d52927-01dd-42da-9f6f-501c00a4110c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:03.661657 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d52927-01dd-42da-9f6f-501c00a4110c None admin] RBAC: Adding query filter params (role.id=66a36c858aab40688080eed6ceace3a0, scope.project.id=16a6348ce8d34348b34d853f76445b3a, user.id=33fab7c55b3a46b5aa91e225ea277804) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:03.662156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d52927-01dd-42da-9f6f-501c00a4110c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:03.663794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d52927-01dd-42da-9f6f-501c00a4110c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:03.669842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 103/205] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:05:03 2022] GET /identity/v3/role_assignments?user.id=33fab7c55b3a46b5aa91e225ea277804&scope.project.id=16a6348ce8d34348b34d853f76445b3a&role.id=66a36c858aab40688080eed6ceace3a0 => generated 259 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:03.674905 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:03.676338 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d52927-01dd-42da-9f6f-501c00a4110c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:03.724315 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:03.779530 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e957de5c-d735-425d-b84b-ef850e4321cb None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:03.810745 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a513e9b-0499-4cdc-9e46-10d9bf1b9c10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:03.811397 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a513e9b-0499-4cdc-9e46-10d9bf1b9c10 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:03.811995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a513e9b-0499-4cdc-9e46-10d9bf1b9c10 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:03.812272 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a513e9b-0499-4cdc-9e46-10d9bf1b9c10 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:03.813541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a513e9b-0499-4cdc-9e46-10d9bf1b9c10 None admin] RBAC: Adding query filter params (role.id=cd616e816a394068867091116a3f1ce0, scope.project.id=16a6348ce8d34348b34d853f76445b3a, user.id=33fab7c55b3a46b5aa91e225ea277804) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:03.813541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a513e9b-0499-4cdc-9e46-10d9bf1b9c10 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:03.816728 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a513e9b-0499-4cdc-9e46-10d9bf1b9c10 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:03.823436 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 103/206] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:05:03 2022] GET /identity/v3/role_assignments?user.id=33fab7c55b3a46b5aa91e225ea277804&scope.project.id=16a6348ce8d34348b34d853f76445b3a&role.id=cd616e816a394068867091116a3f1ce0 => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:03.924400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d52927-01dd-42da-9f6f-501c00a4110c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:03.963754 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc72e34-36b9-4a66-baf1-ddb885e63df1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:03.964763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc72e34-36b9-4a66-baf1-ddb885e63df1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:03.965272 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc72e34-36b9-4a66-baf1-ddb885e63df1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:03.965759 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc72e34-36b9-4a66-baf1-ddb885e63df1 None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:03.967238 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc72e34-36b9-4a66-baf1-ddb885e63df1 None admin] RBAC: Authorizing `identity:get_project(project_id=4426c2e7285b4cbc8331aa86690ef664)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:03.968917 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc72e34-36b9-4a66-baf1-ddb885e63df1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:03.971150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 104/207] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:03 2022] GET /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664 => generated 298 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:03.978877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a513e9b-0499-4cdc-9e46-10d9bf1b9c10 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:04.094409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:04.117517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a513e9b-0499-4cdc-9e46-10d9bf1b9c10 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:04.164217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc78c9fd-967d-44b6-938c-2b3865bd890c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:04.164833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc78c9fd-967d-44b6-938c-2b3865bd890c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:04.165713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc78c9fd-967d-44b6-938c-2b3865bd890c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:04.165991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc78c9fd-967d-44b6-938c-2b3865bd890c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:04.166721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc78c9fd-967d-44b6-938c-2b3865bd890c None admin] RBAC: Adding query filter params (role.id=6f98018cf28149aa84201e6dc4a43c2d, scope.project.id=4426c2e7285b4cbc8331aa86690ef664, user.id=f7c6f07cdd5f4461ae915f6a9d72b7ca) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:04.167054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc78c9fd-967d-44b6-938c-2b3865bd890c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:04.168728 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc78c9fd-967d-44b6-938c-2b3865bd890c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:04.177480 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 104/208] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:05:03 2022] GET /identity/v3/role_assignments?user.id=f7c6f07cdd5f4461ae915f6a9d72b7ca&scope.project.id=4426c2e7285b4cbc8331aa86690ef664&role.id=6f98018cf28149aa84201e6dc4a43c2d => generated 608 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:05.162914 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70677f58-ab98-406f-833b-8295fa987c37 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:05.163436 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70677f58-ab98-406f-833b-8295fa987c37 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:05.163436 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70677f58-ab98-406f-833b-8295fa987c37 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:05.165172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 105/209] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:05.170596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fea84f-1e39-4d1f-aec7-55c119126e53 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:05.171026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fea84f-1e39-4d1f-aec7-55c119126e53 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:05.171195 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fea84f-1e39-4d1f-aec7-55c119126e53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:05.190301 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-43fea84f-1e39-4d1f-aec7-55c119126e53 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:05.191514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-43fea84f-1e39-4d1f-aec7-55c119126e53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:05.265509 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4846a48-a401-4f20-9136-37e47edd83d5 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:05.266344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4846a48-a401-4f20-9136-37e47edd83d5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:05.267067 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4846a48-a401-4f20-9136-37e47edd83d5 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:05.270229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 106/210] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:05 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:05.278643 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ff3613-bd71-46ac-838c-6a6b33949e7d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:05.279946 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 105/211] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:05.280111 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ff3613-bd71-46ac-838c-6a6b33949e7d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:05.280334 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ff3613-bd71-46ac-838c-6a6b33949e7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:05.301955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-d4ff3613-bd71-46ac-838c-6a6b33949e7d None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:05.303181 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d4ff3613-bd71-46ac-838c-6a6b33949e7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:05.373654 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 107/212] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 01:05:05.551284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43fea84f-1e39-4d1f-aec7-55c119126e53 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:05.625694 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:05.652476 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ff3613-bd71-46ac-838c-6a6b33949e7d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:05.659318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43fea84f-1e39-4d1f-aec7-55c119126e53 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:05.668674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:05.696065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937d1153-241e-417d-9d9b-4b858472404d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:05.697333 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937d1153-241e-417d-9d9b-4b858472404d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:05.697612 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937d1153-241e-417d-9d9b-4b858472404d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:05.697884 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937d1153-241e-417d-9d9b-4b858472404d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:05.708567 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937d1153-241e-417d-9d9b-4b858472404d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:05.713725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-937d1153-241e-417d-9d9b-4b858472404d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:05.713725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:05.713725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:05.713725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:05.713725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:05.713725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:05.713725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:05.713725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:05.713725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:05.713725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:05.713725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:05.713725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:05.713725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:05.713725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:05.713725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:05.713725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:05.714587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:05.714587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:05.714587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:05.714587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:05.714587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:05.714587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:05.714587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:05.714587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:05.714587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:05.714587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:05.714587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:05.714587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:05.714587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:05.714587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:05.714587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:05.714587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:05.715408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:05.715408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:05.715408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:05.715408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:05.715408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:05.715408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:05.715408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:05.715408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:05.715408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:05.715408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:05.715408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:05.715408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:05.715408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 106/213] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:05.720688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937d1153-241e-417d-9d9b-4b858472404d None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:05.776436 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ff3613-bd71-46ac-838c-6a6b33949e7d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:05.817131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04db3280-cae4-457d-b552-0bc8de641977 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:05.817898 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04db3280-cae4-457d-b552-0bc8de641977 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:05.818184 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04db3280-cae4-457d-b552-0bc8de641977 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:05.818463 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04db3280-cae4-457d-b552-0bc8de641977 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:05.820259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04db3280-cae4-457d-b552-0bc8de641977 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:05.822100 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04db3280-cae4-457d-b552-0bc8de641977 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:05.826938 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 108/214] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:05.831262 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04db3280-cae4-457d-b552-0bc8de641977 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:05.836519 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937d1153-241e-417d-9d9b-4b858472404d None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:05.870080 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:05.875684 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8029f6ee-de5f-4f11-8a9f-b7c2c9518759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:05.876478 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8029f6ee-de5f-4f11-8a9f-b7c2c9518759 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:05.876939 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8029f6ee-de5f-4f11-8a9f-b7c2c9518759 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:05.877305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8029f6ee-de5f-4f11-8a9f-b7c2c9518759 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:05.878068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8029f6ee-de5f-4f11-8a9f-b7c2c9518759 None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:05.878486 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8029f6ee-de5f-4f11-8a9f-b7c2c9518759 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:05.880203 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8029f6ee-de5f-4f11-8a9f-b7c2c9518759 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:05.893907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 107/215] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:05.895040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8029f6ee-de5f-4f11-8a9f-b7c2c9518759 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:05.941256 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:05.968406 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04db3280-cae4-457d-b552-0bc8de641977 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:06.004056 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8469f48d-23c0-4141-bedc-9bb2e20b1fde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:06.005158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8469f48d-23c0-4141-bedc-9bb2e20b1fde None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:06.005641 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8469f48d-23c0-4141-bedc-9bb2e20b1fde None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:06.006143 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8469f48d-23c0-4141-bedc-9bb2e20b1fde None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:06.007097 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8469f48d-23c0-4141-bedc-9bb2e20b1fde None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:06.009087 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8469f48d-23c0-4141-bedc-9bb2e20b1fde None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:06.015096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8029f6ee-de5f-4f11-8a9f-b7c2c9518759 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:06.037882 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 109/216] 10.222.0.15 () {60 vars in 1132 bytes} [Tue Sep 13 01:05:05 2022] POST /identity/v3/users => generated 297 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:05:06.076657 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59661883-3cd6-42d6-9cb1-53420f37322b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:06.077761 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59661883-3cd6-42d6-9cb1-53420f37322b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:06.078242 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59661883-3cd6-42d6-9cb1-53420f37322b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:06.078714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59661883-3cd6-42d6-9cb1-53420f37322b None admin] PATH_INFO: `/v3/users/6a22d9a5c1634cdb81a5c8e05b18a550` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:06.095413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59661883-3cd6-42d6-9cb1-53420f37322b None admin] RBAC: Authorizing `identity:get_user(user_id=6a22d9a5c1634cdb81a5c8e05b18a550)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:06.097259 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59661883-3cd6-42d6-9cb1-53420f37322b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:06.099389 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 108/217] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:05 2022] GET /identity/v3/users/6a22d9a5c1634cdb81a5c8e05b18a550 => generated 297 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:06.112114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8469f48d-23c0-4141-bedc-9bb2e20b1fde None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:06.129026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b872cf80-1084-40ed-8dc5-5c4125efbb9a None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:06.129342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b872cf80-1084-40ed-8dc5-5c4125efbb9a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:06.129581 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b872cf80-1084-40ed-8dc5-5c4125efbb9a None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:06.130990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 109/218] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:05:06.137102 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae13e3f-a8e6-4f88-8df3-a9dc070ca024 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:06.137402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae13e3f-a8e6-4f88-8df3-a9dc070ca024 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:06.137649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae13e3f-a8e6-4f88-8df3-a9dc070ca024 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:06.157061 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-cae13e3f-a8e6-4f88-8df3-a9dc070ca024 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:06.158267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cae13e3f-a8e6-4f88-8df3-a9dc070ca024 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:06.219002 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:06.237860 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8469f48d-23c0-4141-bedc-9bb2e20b1fde None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:06.247766 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 110/219] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 01:05:06.281595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e0ca81-1911-4a5c-ab8c-c146d80ee605 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:06.281595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e0ca81-1911-4a5c-ab8c-c146d80ee605 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:06.281595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e0ca81-1911-4a5c-ab8c-c146d80ee605 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:06.281595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e0ca81-1911-4a5c-ab8c-c146d80ee605 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:06.286690 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:06.307267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62455f5-c2e2-40e5-b809-840ed05a3ea3 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:06.308186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62455f5-c2e2-40e5-b809-840ed05a3ea3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:06.308631 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62455f5-c2e2-40e5-b809-840ed05a3ea3 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:06.310130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 111/220] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:05:06.316543 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c54c105-9934-4461-819d-33a5c63d1d45 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:06.317504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c54c105-9934-4461-819d-33a5c63d1d45 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:06.317955 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c54c105-9934-4461-819d-33a5c63d1d45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:06.340572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7c54c105-9934-4461-819d-33a5c63d1d45 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:06.341672 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7c54c105-9934-4461-819d-33a5c63d1d45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:06.364656 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e0ca81-1911-4a5c-ab8c-c146d80ee605 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:06.366163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e0ca81-1911-4a5c-ab8c-c146d80ee605 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:06.427757 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 112/221] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 01:05:06.483579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 110/222] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 13 01:05:06.488901 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c54c105-9934-4461-819d-33a5c63d1d45 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:06.523225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e0ca81-1911-4a5c-ab8c-c146d80ee605 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:06.572130 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:06.603476 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c54c105-9934-4461-819d-33a5c63d1d45 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:06.619804 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:06.644727 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e0ca81-1911-4a5c-ab8c-c146d80ee605 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:06.661814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6621f4-2964-45be-8f75-f2fa178b9a66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:06.662847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6621f4-2964-45be-8f75-f2fa178b9a66 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:06.663422 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6621f4-2964-45be-8f75-f2fa178b9a66 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:06.664068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6621f4-2964-45be-8f75-f2fa178b9a66 None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:06.667818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6621f4-2964-45be-8f75-f2fa178b9a66 None admin] RBAC: Authorizing `identity:get_project(project_id=16a6348ce8d34348b34d853f76445b3a)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:06.670470 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6621f4-2964-45be-8f75-f2fa178b9a66 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:06.673563 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 113/223] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:06 2022] GET /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a => generated 302 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:06.682919 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6621f4-2964-45be-8f75-f2fa178b9a66 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:06.685959 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd30ef58-c1fe-475a-905b-659b49f0c05d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:06.686799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd30ef58-c1fe-475a-905b-659b49f0c05d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:06.687241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd30ef58-c1fe-475a-905b-659b49f0c05d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:06.687654 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd30ef58-c1fe-475a-905b-659b49f0c05d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:06.700439 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd30ef58-c1fe-475a-905b-659b49f0c05d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:06.706391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-dd30ef58-c1fe-475a-905b-659b49f0c05d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:05:06.706391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:06.706391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:06.706391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:06.706391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:06.706391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:06.706391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:06.706391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:06.706391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:06.706391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:06.706391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:06.706391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:06.706391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:06.706391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:06.706391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:06.706391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:06.707428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:06.707428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:06.707428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:06.707428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:06.707428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:06.707428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:06.707428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:06.707428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:06.707428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:06.707428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:06.707428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:06.707428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:06.707428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:06.707428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:06.707428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:06.707428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:06.708414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:06.708414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:06.708414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:06.708414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:06.708414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:06.708414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:06.708414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:06.708414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:06.708414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:06.708414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:06.708414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:05:06.708414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:06.710442 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 111/224] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:05:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:05:06.717368 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd30ef58-c1fe-475a-905b-659b49f0c05d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:06.830887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:06.842877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd30ef58-c1fe-475a-905b-659b49f0c05d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:06.871863 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6621f4-2964-45be-8f75-f2fa178b9a66 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:06.890631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b5fcb7-73b6-4037-b08b-edbee3e59f96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:06.891272 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b5fcb7-73b6-4037-b08b-edbee3e59f96 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:06.891545 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b5fcb7-73b6-4037-b08b-edbee3e59f96 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:06.891810 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b5fcb7-73b6-4037-b08b-edbee3e59f96 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:06.892465 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b5fcb7-73b6-4037-b08b-edbee3e59f96 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:06.892763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b5fcb7-73b6-4037-b08b-edbee3e59f96 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:06.894357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b5fcb7-73b6-4037-b08b-edbee3e59f96 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:06.905088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 112/225] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:06.905709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b5fcb7-73b6-4037-b08b-edbee3e59f96 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:06.908592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8920c0c6-0e87-49c8-958d-cc10482cbedc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:06.909361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8920c0c6-0e87-49c8-958d-cc10482cbedc None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:06.909638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8920c0c6-0e87-49c8-958d-cc10482cbedc None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:06.909910 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8920c0c6-0e87-49c8-958d-cc10482cbedc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:06.910639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8920c0c6-0e87-49c8-958d-cc10482cbedc None admin] RBAC: Adding query filter params (role.id=6f71034643cb4e8caee299f5d0bb863f, scope.project.id=16a6348ce8d34348b34d853f76445b3a, user.id=6a22d9a5c1634cdb81a5c8e05b18a550) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:06.910972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8920c0c6-0e87-49c8-958d-cc10482cbedc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:06.912706 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8920c0c6-0e87-49c8-958d-cc10482cbedc None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:06.919463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 114/226] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:05:06 2022] GET /identity/v3/role_assignments?user.id=6a22d9a5c1634cdb81a5c8e05b18a550&scope.project.id=16a6348ce8d34348b34d853f76445b3a&role.id=6f71034643cb4e8caee299f5d0bb863f => generated 259 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:06.920256 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8920c0c6-0e87-49c8-958d-cc10482cbedc None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:07.011253 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:07.068208 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b5fcb7-73b6-4037-b08b-edbee3e59f96 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:07.113728 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8920c0c6-0e87-49c8-958d-cc10482cbedc None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:07.127747 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c98b96b-c24a-49ff-8e61-f67348a79266 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:07.129438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c98b96b-c24a-49ff-8e61-f67348a79266 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:07.129921 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c98b96b-c24a-49ff-8e61-f67348a79266 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:07.130667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c98b96b-c24a-49ff-8e61-f67348a79266 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:07.145884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c98b96b-c24a-49ff-8e61-f67348a79266 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:07.153484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7726b3-f9ee-453a-a43d-ffef931a1c4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:07.153804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-8c98b96b-c24a-49ff-8e61-f67348a79266 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:07.153804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:07.153804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:07.153804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:07.153804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:07.153804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:07.153804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:07.153804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:07.153804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:07.153804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:07.153804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:07.153804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:07.153804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:07.153804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:07.153804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:07.153804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:07.154653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:07.154653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:07.154653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:07.154653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:07.154653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:07.154653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:07.154653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:07.154653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:07.154653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:07.154653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:07.154653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:07.154653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:07.154653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:07.154653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:07.154653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:07.154653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:07.155472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:07.155472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:07.155472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:07.155472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:07.155472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:07.155472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:07.155472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:07.155472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:07.155472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:07.155472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:07.155472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:07.155472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:07.156050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7726b3-f9ee-453a-a43d-ffef931a1c4b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:07.156050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7726b3-f9ee-453a-a43d-ffef931a1c4b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:07.156050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7726b3-f9ee-453a-a43d-ffef931a1c4b None admin] PATH_INFO: `/v3/users/33fab7c55b3a46b5aa91e225ea277804` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:07.156902 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7726b3-f9ee-453a-a43d-ffef931a1c4b None admin] RBAC: Authorizing `identity:get_user(user_id=33fab7c55b3a46b5aa91e225ea277804)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:07.158195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 113/227] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:05:06 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:07.158385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7726b3-f9ee-453a-a43d-ffef931a1c4b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:07.162898 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 115/228] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:06 2022] GET /identity/v3/users/33fab7c55b3a46b5aa91e225ea277804 => generated 283 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:07.174975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c98b96b-c24a-49ff-8e61-f67348a79266 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:07.175137 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7726b3-f9ee-453a-a43d-ffef931a1c4b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:07.188851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:07.277952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c98b96b-c24a-49ff-8e61-f67348a79266 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:07.314868 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53da4f99-8d37-4258-b72c-c0348577c816 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:07.315623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53da4f99-8d37-4258-b72c-c0348577c816 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:07.316147 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53da4f99-8d37-4258-b72c-c0348577c816 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:07.316493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53da4f99-8d37-4258-b72c-c0348577c816 None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:07.318265 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53da4f99-8d37-4258-b72c-c0348577c816 None admin] RBAC: Authorizing `identity:get_project(project_id=16a6348ce8d34348b34d853f76445b3a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:07.319799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53da4f99-8d37-4258-b72c-c0348577c816 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:07.321806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 114/229] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:07 2022] GET /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a => generated 302 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:07.327349 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53da4f99-8d37-4258-b72c-c0348577c816 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:07.427669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:07.443584 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53da4f99-8d37-4258-b72c-c0348577c816 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:07.458501 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7726b3-f9ee-453a-a43d-ffef931a1c4b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:07.485934 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:07.499911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f674eaf3-cd02-44da-a8b3-bc71e778df47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:07.502018 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f674eaf3-cd02-44da-a8b3-bc71e778df47 None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:07.502302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f674eaf3-cd02-44da-a8b3-bc71e778df47 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:07.502578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f674eaf3-cd02-44da-a8b3-bc71e778df47 None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a/users/33fab7c55b3a46b5aa91e225ea277804/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:07.507070 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f674eaf3-cd02-44da-a8b3-bc71e778df47 None admin] RBAC: Authorizing `identity:create_grant(project_id=16a6348ce8d34348b34d853f76445b3a, user_id=33fab7c55b3a46b5aa91e225ea277804, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:07.509379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f674eaf3-cd02-44da-a8b3-bc71e778df47 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:07.525194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df23442-de4e-4b94-8996-870c92cd3efd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:07.532960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 115/230] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:05:07 2022] PUT /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a/users/33fab7c55b3a46b5aa91e225ea277804/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:05:07.533209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df23442-de4e-4b94-8996-870c92cd3efd None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:07.533630 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df23442-de4e-4b94-8996-870c92cd3efd None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:07.533910 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df23442-de4e-4b94-8996-870c92cd3efd None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:07.534723 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df23442-de4e-4b94-8996-870c92cd3efd None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:07.535097 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df23442-de4e-4b94-8996-870c92cd3efd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:07.536699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df23442-de4e-4b94-8996-870c92cd3efd None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:07.547447 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 116/231] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:07 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 379 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:07.559891 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f674eaf3-cd02-44da-a8b3-bc71e778df47 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:07.682162 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f674eaf3-cd02-44da-a8b3-bc71e778df47 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:07.719847 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b39b351-5f62-4600-bae3-87dba778c178 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:07.720563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b39b351-5f62-4600-bae3-87dba778c178 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:07.720841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b39b351-5f62-4600-bae3-87dba778c178 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:07.721114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b39b351-5f62-4600-bae3-87dba778c178 None admin] PATH_INFO: `/v3/users/33fab7c55b3a46b5aa91e225ea277804` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:07.722215 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b39b351-5f62-4600-bae3-87dba778c178 None admin] RBAC: Authorizing `identity:get_user(user_id=33fab7c55b3a46b5aa91e225ea277804)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:07.723670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b39b351-5f62-4600-bae3-87dba778c178 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:07.725530 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 116/232] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:07 2022] GET /identity/v3/users/33fab7c55b3a46b5aa91e225ea277804 => generated 283 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:07.730799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df23442-de4e-4b94-8996-870c92cd3efd None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:07.840133 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df23442-de4e-4b94-8996-870c92cd3efd None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:07.894225 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11cc6ee-2335-4550-8bfd-45680d44089d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:07.894873 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11cc6ee-2335-4550-8bfd-45680d44089d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:07.895157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11cc6ee-2335-4550-8bfd-45680d44089d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:07.895425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11cc6ee-2335-4550-8bfd-45680d44089d None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:07.899797 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11cc6ee-2335-4550-8bfd-45680d44089d None admin] RBAC: Authorizing `identity:get_project(project_id=16a6348ce8d34348b34d853f76445b3a)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:07.901247 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11cc6ee-2335-4550-8bfd-45680d44089d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:07.903242 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 117/233] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:07 2022] GET /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a => generated 302 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:07.908513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b39b351-5f62-4600-bae3-87dba778c178 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:07.935815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:08.042696 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b39b351-5f62-4600-bae3-87dba778c178 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:08.078005 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474c51df-000c-456f-8a65-8b87ab9fa53e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:08.078675 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474c51df-000c-456f-8a65-8b87ab9fa53e None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:08.078974 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474c51df-000c-456f-8a65-8b87ab9fa53e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:08.079242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474c51df-000c-456f-8a65-8b87ab9fa53e None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a/users/33fab7c55b3a46b5aa91e225ea277804/roles/cd616e816a394068867091116a3f1ce0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:08.081367 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474c51df-000c-456f-8a65-8b87ab9fa53e None admin] RBAC: Authorizing `identity:create_grant(project_id=16a6348ce8d34348b34d853f76445b3a, user_id=33fab7c55b3a46b5aa91e225ea277804, role_id=cd616e816a394068867091116a3f1ce0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:08.082814 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474c51df-000c-456f-8a65-8b87ab9fa53e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:08.100651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 117/234] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:05:07 2022] PUT /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a/users/33fab7c55b3a46b5aa91e225ea277804/roles/cd616e816a394068867091116a3f1ce0 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:05:08.994585 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717020ca-0983-4486-b3e3-61bd68089a72 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:08.995141 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717020ca-0983-4486-b3e3-61bd68089a72 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:08.995141 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717020ca-0983-4486-b3e3-61bd68089a72 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:08.996787 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 118/235] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:09.002336 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560a5ffc-4184-4803-8686-984c676f1e74 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:09.003298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560a5ffc-4184-4803-8686-984c676f1e74 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:09.003768 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560a5ffc-4184-4803-8686-984c676f1e74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:09.030171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-560a5ffc-4184-4803-8686-984c676f1e74 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:09.031880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-560a5ffc-4184-4803-8686-984c676f1e74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:09.130620 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 118/236] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:09.209378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab406855-701d-425b-a28b-0767e7f7b74e None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:09.210002 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab406855-701d-425b-a28b-0767e7f7b74e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:09.210475 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab406855-701d-425b-a28b-0767e7f7b74e None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:09.212147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 119/237] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:09.218158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898edc93-8695-4e17-af8b-5e064e323daa None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:09.218711 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898edc93-8695-4e17-af8b-5e064e323daa None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:09.219209 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898edc93-8695-4e17-af8b-5e064e323daa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:09.231562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:09.246881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-898edc93-8695-4e17-af8b-5e064e323daa None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:09.248002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-898edc93-8695-4e17-af8b-5e064e323daa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:09.340870 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 119/238] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:09.467846 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab406855-701d-425b-a28b-0767e7f7b74e None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:09.540130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:09.617825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab406855-701d-425b-a28b-0767e7f7b74e None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:09.637959 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7e07c3-916f-4ef4-9107-dcd811787402 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:09.638555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7e07c3-916f-4ef4-9107-dcd811787402 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:09.639073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7e07c3-916f-4ef4-9107-dcd811787402 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:09.640668 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 120/239] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:05:09.652876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffb48e4-df8f-404d-b65e-c7f7d2995579 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:09.653479 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffb48e4-df8f-404d-b65e-c7f7d2995579 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:09.653971 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffb48e4-df8f-404d-b65e-c7f7d2995579 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:09.659302 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6f7f8c-b80c-49c2-8af6-a3ef010b1f62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:09.659934 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6f7f8c-b80c-49c2-8af6-a3ef010b1f62 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:09.660292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6f7f8c-b80c-49c2-8af6-a3ef010b1f62 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:09.660524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6f7f8c-b80c-49c2-8af6-a3ef010b1f62 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:09.683551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6f7f8c-b80c-49c2-8af6-a3ef010b1f62 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:09.689157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-2a6f7f8c-b80c-49c2-8af6-a3ef010b1f62 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 01:05:09.689157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:09.689157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:09.689157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:09.689157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:09.689157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:09.689157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:09.689157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:09.689157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:09.689157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:09.689157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:09.689157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:09.689157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:09.689157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:09.689157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:09.689157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:09.690031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:09.690031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:09.690031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:09.690031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:09.690031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:09.690031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:09.690031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:09.690031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:09.690031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:09.690031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:09.690031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:09.690031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:09.690031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:09.690031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:09.690031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:09.690031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:09.690806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:09.690806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:09.690806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:09.690806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:09.690806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:09.690806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:09.690806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:09.690806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:09.690806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:09.690806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:09.690806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 01:05:09.690806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:09.694742 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8ffb48e4-df8f-404d-b65e-c7f7d2995579 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:09.694977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 120/240] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:09 2022] GET /identity/v3/roles/reader => generated 84 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:09.696350 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8ffb48e4-df8f-404d-b65e-c7f7d2995579 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:09.703808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6f7f8c-b80c-49c2-8af6-a3ef010b1f62 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:09.771935 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 121/241] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:09.773076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffb48e4-df8f-404d-b65e-c7f7d2995579 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:09.836227 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:09.863797 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6f7f8c-b80c-49c2-8af6-a3ef010b1f62 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:09.865340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:09.876293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffb48e4-df8f-404d-b65e-c7f7d2995579 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:09.916497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9864fd90-c181-44e1-b762-624a77256aa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:09.917618 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9864fd90-c181-44e1-b762-624a77256aa9 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:09.918127 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9864fd90-c181-44e1-b762-624a77256aa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:09.918647 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9864fd90-c181-44e1-b762-624a77256aa9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:09.923237 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaca00a-1653-4057-a7fc-53460d48e1d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:09.923571 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9864fd90-c181-44e1-b762-624a77256aa9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:09.923848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaca00a-1653-4057-a7fc-53460d48e1d6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:09.924139 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaca00a-1653-4057-a7fc-53460d48e1d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:09.924410 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaca00a-1653-4057-a7fc-53460d48e1d6 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:09.925021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9864fd90-c181-44e1-b762-624a77256aa9 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:09.925125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaca00a-1653-4057-a7fc-53460d48e1d6 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:09.925362 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaca00a-1653-4057-a7fc-53460d48e1d6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:09.926727 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaca00a-1653-4057-a7fc-53460d48e1d6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:09.926925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 122/242] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:09.932308 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9864fd90-c181-44e1-b762-624a77256aa9 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:09.947733 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 121/243] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:09 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:09.959849 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaca00a-1653-4057-a7fc-53460d48e1d6 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:10.079897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9864fd90-c181-44e1-b762-624a77256aa9 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:10.127762 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:10.132997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26bd3fed-7011-4917-91f5-8771cfb91c01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:10.134153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bd3fed-7011-4917-91f5-8771cfb91c01 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:10.134440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bd3fed-7011-4917-91f5-8771cfb91c01 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:10.134708 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bd3fed-7011-4917-91f5-8771cfb91c01 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:10.135404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26bd3fed-7011-4917-91f5-8771cfb91c01 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:10.137126 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26bd3fed-7011-4917-91f5-8771cfb91c01 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:10.163041 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 123/244] 10.222.0.15 () {60 vars in 1132 bytes} [Tue Sep 13 01:05:09 2022] POST /identity/v3/users => generated 293 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:05:10.164200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26bd3fed-7011-4917-91f5-8771cfb91c01 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:10.186724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaca00a-1653-4057-a7fc-53460d48e1d6 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:10.268892 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1715a5c-bec1-4847-afda-d3742817214c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:10.269788 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1715a5c-bec1-4847-afda-d3742817214c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:10.270211 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1715a5c-bec1-4847-afda-d3742817214c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:10.270651 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1715a5c-bec1-4847-afda-d3742817214c None admin] PATH_INFO: `/v3/users/b028bffe18854cc9b2980b8c005aac4d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:10.287725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1715a5c-bec1-4847-afda-d3742817214c None admin] RBAC: Authorizing `identity:get_user(user_id=b028bffe18854cc9b2980b8c005aac4d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:10.287924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26bd3fed-7011-4917-91f5-8771cfb91c01 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:10.289891 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1715a5c-bec1-4847-afda-d3742817214c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:10.293407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 122/245] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:09 2022] GET /identity/v3/users/b028bffe18854cc9b2980b8c005aac4d => generated 297 bytes in 334 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:10.295071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122dfae4-4018-4f18-807b-4c6366ac9ac2 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:10.295579 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122dfae4-4018-4f18-807b-4c6366ac9ac2 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:10.296028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122dfae4-4018-4f18-807b-4c6366ac9ac2 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:10.298133 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 123/246] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:10.303235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122dfae4-4018-4f18-807b-4c6366ac9ac2 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:10.334777 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:10.336415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9f8dd3-b5f0-46fe-b5ce-a2140fa8011b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:10.337999 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9f8dd3-b5f0-46fe-b5ce-a2140fa8011b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:10.338669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9f8dd3-b5f0-46fe-b5ce-a2140fa8011b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:10.339184 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9f8dd3-b5f0-46fe-b5ce-a2140fa8011b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:10.351761 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9f8dd3-b5f0-46fe-b5ce-a2140fa8011b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:10.358228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-8f9f8dd3-b5f0-46fe-b5ce-a2140fa8011b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:10.358228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:10.358228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:10.358228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:10.358228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:10.358228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:10.358228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:10.358228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:10.358228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:10.358228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:10.358228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:10.358228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:10.358228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:10.358228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:10.358228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:10.358228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:10.359175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:10.359175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:10.359175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:10.359175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:10.359175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:10.359175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:10.359175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:10.359175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:10.359175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:10.359175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:10.359175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:10.359175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:10.359175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:10.359175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:10.359175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:10.359175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:10.359968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:10.359968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:10.359968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:10.359968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:10.359968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:10.359968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:10.359968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:10.359968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:10.359968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:10.359968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:10.359968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:10.359968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:10.361230 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 124/247] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:10.361319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ed71e6-1ad5-408c-9ef5-047ad30b98e8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:10.361588 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ed71e6-1ad5-408c-9ef5-047ad30b98e8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:10.361859 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ed71e6-1ad5-408c-9ef5-047ad30b98e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:10.383059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-58ed71e6-1ad5-408c-9ef5-047ad30b98e8 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:10.384972 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-58ed71e6-1ad5-408c-9ef5-047ad30b98e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:10.451083 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122dfae4-4018-4f18-807b-4c6366ac9ac2 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:10.452130 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:10.481266 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 125/248] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:10.487941 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ed71e6-1ad5-408c-9ef5-047ad30b98e8 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:10.491741 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed46a5d6-5c92-41dc-861e-3504b502852b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:10.492572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed46a5d6-5c92-41dc-861e-3504b502852b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:10.493605 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed46a5d6-5c92-41dc-861e-3504b502852b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:10.494091 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed46a5d6-5c92-41dc-861e-3504b502852b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:10.576042 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed46a5d6-5c92-41dc-861e-3504b502852b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:10.577898 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed46a5d6-5c92-41dc-861e-3504b502852b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:10.606049 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:10.663183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ed71e6-1ad5-408c-9ef5-047ad30b98e8 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:10.694930 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 124/249] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:10.699580 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed46a5d6-5c92-41dc-861e-3504b502852b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:10.702869 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea1387a-7de2-4ce4-abe8-de1b98e49d97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:10.703753 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea1387a-7de2-4ce4-abe8-de1b98e49d97 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:10.704234 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea1387a-7de2-4ce4-abe8-de1b98e49d97 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:10.704707 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea1387a-7de2-4ce4-abe8-de1b98e49d97 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:10.705692 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea1387a-7de2-4ce4-abe8-de1b98e49d97 None admin] RBAC: Adding query filter params (name=member) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:10.706193 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea1387a-7de2-4ce4-abe8-de1b98e49d97 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:10.707812 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea1387a-7de2-4ce4-abe8-de1b98e49d97 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:10.718288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 126/250] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:10.724077 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea1387a-7de2-4ce4-abe8-de1b98e49d97 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:10.776907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:10.842271 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed46a5d6-5c92-41dc-861e-3504b502852b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:10.872704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea1387a-7de2-4ce4-abe8-de1b98e49d97 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:10.873716 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:10.888443 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608197ab-6da1-4ee0-8c61-6b2eba359bad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:10.889085 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608197ab-6da1-4ee0-8c61-6b2eba359bad None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:10.889361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608197ab-6da1-4ee0-8c61-6b2eba359bad None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:10.889631 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608197ab-6da1-4ee0-8c61-6b2eba359bad None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:10.891009 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608197ab-6da1-4ee0-8c61-6b2eba359bad None admin] RBAC: Authorizing `identity:get_project(project_id=16a6348ce8d34348b34d853f76445b3a)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:10.892465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608197ab-6da1-4ee0-8c61-6b2eba359bad None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:10.894495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 125/251] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:10 2022] GET /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a => generated 302 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:10.900514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608197ab-6da1-4ee0-8c61-6b2eba359bad None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:10.918309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c7368d-b8cb-4c6e-b7d6-c4e03b2c5d31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:10.919808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c7368d-b8cb-4c6e-b7d6-c4e03b2c5d31 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:10.920324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c7368d-b8cb-4c6e-b7d6-c4e03b2c5d31 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:10.920828 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c7368d-b8cb-4c6e-b7d6-c4e03b2c5d31 None admin] PATH_INFO: `/v3/users/6a22d9a5c1634cdb81a5c8e05b18a550` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:10.922171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c7368d-b8cb-4c6e-b7d6-c4e03b2c5d31 None admin] RBAC: Authorizing `identity:get_user(user_id=6a22d9a5c1634cdb81a5c8e05b18a550)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:10.924272 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c7368d-b8cb-4c6e-b7d6-c4e03b2c5d31 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:10.926144 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 127/252] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:10 2022] GET /identity/v3/users/6a22d9a5c1634cdb81a5c8e05b18a550 => generated 297 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:10.931817 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c7368d-b8cb-4c6e-b7d6-c4e03b2c5d31 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:11.058099 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c7368d-b8cb-4c6e-b7d6-c4e03b2c5d31 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:11.068939 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:11.085590 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608197ab-6da1-4ee0-8c61-6b2eba359bad None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:11.097200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1413b14b-8c8a-4b12-914f-586b9cd5b3c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:11.097902 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1413b14b-8c8a-4b12-914f-586b9cd5b3c5 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:11.098181 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1413b14b-8c8a-4b12-914f-586b9cd5b3c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:11.098448 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1413b14b-8c8a-4b12-914f-586b9cd5b3c5 None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:11.099611 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1413b14b-8c8a-4b12-914f-586b9cd5b3c5 None admin] RBAC: Authorizing `identity:get_project(project_id=16a6348ce8d34348b34d853f76445b3a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:11.101050 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1413b14b-8c8a-4b12-914f-586b9cd5b3c5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:11.108418 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 128/253] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:10 2022] GET /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a => generated 302 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:11.108418 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1413b14b-8c8a-4b12-914f-586b9cd5b3c5 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:11.142506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ff0f5e-7ff9-464a-bff8-dd48e95797d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:11.143138 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ff0f5e-7ff9-464a-bff8-dd48e95797d2 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:11.143657 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ff0f5e-7ff9-464a-bff8-dd48e95797d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:11.143932 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ff0f5e-7ff9-464a-bff8-dd48e95797d2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:11.145069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ff0f5e-7ff9-464a-bff8-dd48e95797d2 None admin] RBAC: Adding query filter params (role.id=6f98018cf28149aa84201e6dc4a43c2d, scope.project.id=16a6348ce8d34348b34d853f76445b3a, user.id=b028bffe18854cc9b2980b8c005aac4d) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:11.145636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ff0f5e-7ff9-464a-bff8-dd48e95797d2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:11.147094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ff0f5e-7ff9-464a-bff8-dd48e95797d2 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:11.158956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 126/254] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:05:10 2022] GET /identity/v3/role_assignments?user.id=b028bffe18854cc9b2980b8c005aac4d&scope.project.id=16a6348ce8d34348b34d853f76445b3a&role.id=6f98018cf28149aa84201e6dc4a43c2d => generated 259 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:11.160422 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf012ba-919c-4bc3-a001-3e3cad48eda9 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:11.160707 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf012ba-919c-4bc3-a001-3e3cad48eda9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:11.160981 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf012ba-919c-4bc3-a001-3e3cad48eda9 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:11.162775 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 127/255] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:11.165401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf012ba-919c-4bc3-a001-3e3cad48eda9 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:11.251222 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:11.305025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf012ba-919c-4bc3-a001-3e3cad48eda9 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:11.325159 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1413b14b-8c8a-4b12-914f-586b9cd5b3c5 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:11.364059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39f2435-3796-4455-bfd9-77afc3f7bd07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:11.364735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39f2435-3796-4455-bfd9-77afc3f7bd07 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:11.365049 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39f2435-3796-4455-bfd9-77afc3f7bd07 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:11.365331 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39f2435-3796-4455-bfd9-77afc3f7bd07 None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a/users/6a22d9a5c1634cdb81a5c8e05b18a550/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:11.368720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39f2435-3796-4455-bfd9-77afc3f7bd07 None admin] RBAC: Authorizing `identity:create_grant(project_id=16a6348ce8d34348b34d853f76445b3a, user_id=6a22d9a5c1634cdb81a5c8e05b18a550, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:11.370172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39f2435-3796-4455-bfd9-77afc3f7bd07 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:11.385938 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca323b9d-a4a3-441d-8dd6-2738ab1b7086 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:11.387004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca323b9d-a4a3-441d-8dd6-2738ab1b7086 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:11.387501 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca323b9d-a4a3-441d-8dd6-2738ab1b7086 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:11.387959 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca323b9d-a4a3-441d-8dd6-2738ab1b7086 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:11.389451 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 128/256] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:05:11 2022] PUT /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a/users/6a22d9a5c1634cdb81a5c8e05b18a550/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:05:11.391124 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b15a01b-d7a0-4312-b6ac-bcdc65a4fcf2 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:11.391499 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b15a01b-d7a0-4312-b6ac-bcdc65a4fcf2 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:11.391781 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b15a01b-d7a0-4312-b6ac-bcdc65a4fcf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:11.411819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca323b9d-a4a3-441d-8dd6-2738ab1b7086 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:11.418811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-ca323b9d-a4a3-441d-8dd6-2738ab1b7086 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:05:11.418811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:11.418811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:11.418811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:11.418811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:11.418811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:11.418811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:11.418811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:11.418811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:11.418811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:11.418811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:11.418811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:11.418811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:11.418811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:11.418811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:11.418811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:11.419790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:11.419790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:11.419790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:11.419790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:11.419790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:11.419790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:11.419790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:11.419790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:11.419790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:11.419790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:11.419790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:11.419790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:11.419790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:11.419790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:11.419790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:11.419790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:11.420591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:11.420591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:11.420591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:11.420591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:11.420591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:11.420591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:11.420591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:11.420591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:11.420591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:11.420591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:11.420591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:05:11.420591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:11.426257 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 129/257] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:05:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 322 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:11.431434 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7b15a01b-d7a0-4312-b6ac-bcdc65a4fcf2 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:11.433079 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7b15a01b-d7a0-4312-b6ac-bcdc65a4fcf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:11.434459 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca323b9d-a4a3-441d-8dd6-2738ab1b7086 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:11.516561 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:11.526662 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 129/258] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:11.594482 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca323b9d-a4a3-441d-8dd6-2738ab1b7086 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:11.632383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95894d5-e018-4cb6-bd17-dbd254557e04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:11.633018 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95894d5-e018-4cb6-bd17-dbd254557e04 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:11.633297 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95894d5-e018-4cb6-bd17-dbd254557e04 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:11.634098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95894d5-e018-4cb6-bd17-dbd254557e04 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:11.634786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95894d5-e018-4cb6-bd17-dbd254557e04 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:11.635128 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95894d5-e018-4cb6-bd17-dbd254557e04 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:11.636563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95894d5-e018-4cb6-bd17-dbd254557e04 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:11.649821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 130/259] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:11.653083 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b15a01b-d7a0-4312-b6ac-bcdc65a4fcf2 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:11.704147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:11.773798 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b15a01b-d7a0-4312-b6ac-bcdc65a4fcf2 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:11.823830 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4f8dca-831d-40fa-b261-0423d447c876 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:11.824795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4f8dca-831d-40fa-b261-0423d447c876 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:11.825252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4f8dca-831d-40fa-b261-0423d447c876 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:11.825768 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4f8dca-831d-40fa-b261-0423d447c876 None admin] PATH_INFO: `/v3/users/33fab7c55b3a46b5aa91e225ea277804` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:11.831802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4f8dca-831d-40fa-b261-0423d447c876 None admin] RBAC: Authorizing `identity:get_user(user_id=33fab7c55b3a46b5aa91e225ea277804)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:11.833763 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4f8dca-831d-40fa-b261-0423d447c876 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:11.836124 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 130/260] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:11 2022] GET /identity/v3/users/33fab7c55b3a46b5aa91e225ea277804 => generated 283 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:11.841998 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95894d5-e018-4cb6-bd17-dbd254557e04 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:11.970773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:11.975397 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95894d5-e018-4cb6-bd17-dbd254557e04 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:12.023566 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4464d3-899c-42ab-ad0c-d64cb6f41b78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:12.024852 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4464d3-899c-42ab-ad0c-d64cb6f41b78 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:12.025135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4464d3-899c-42ab-ad0c-d64cb6f41b78 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:12.025346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4464d3-899c-42ab-ad0c-d64cb6f41b78 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:12.114188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4464d3-899c-42ab-ad0c-d64cb6f41b78 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:12.115656 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4464d3-899c-42ab-ad0c-d64cb6f41b78 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:12.151767 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4f8dca-831d-40fa-b261-0423d447c876 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:12.267795 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:12.277691 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 131/261] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:12.285067 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4464d3-899c-42ab-ad0c-d64cb6f41b78 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:12.325573 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4f8dca-831d-40fa-b261-0423d447c876 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:12.340514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:12.371555 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7ff8fb-eb83-4dfb-bcb3-5e85900a4195 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:12.372307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7ff8fb-eb83-4dfb-bcb3-5e85900a4195 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:12.372587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7ff8fb-eb83-4dfb-bcb3-5e85900a4195 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:12.372858 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7ff8fb-eb83-4dfb-bcb3-5e85900a4195 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:12.395983 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7ff8fb-eb83-4dfb-bcb3-5e85900a4195 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:12.401094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-2a7ff8fb-eb83-4dfb-bcb3-5e85900a4195 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:12.401094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:12.401094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:12.401094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:12.401094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:12.401094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:12.401094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:12.401094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:12.401094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:12.401094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:12.401094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:12.401094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:12.401094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:12.401094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:12.401094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:12.401094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:12.402025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:12.402025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:12.402025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:12.402025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:12.402025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:12.402025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:12.402025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:12.402025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:12.402025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:12.402025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:12.402025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:12.402025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:12.402025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:12.402025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:12.402025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:12.402025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:12.403465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:12.403465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:12.403465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:12.403465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:12.403465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:12.403465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:12.403465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:12.403465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:12.403465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:12.403465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:12.403465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:12.403465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:12.403465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 131/262] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:05:12 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:12.408111 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7ff8fb-eb83-4dfb-bcb3-5e85900a4195 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:12.458427 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4464d3-899c-42ab-ad0c-d64cb6f41b78 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:12.547642 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60a0ff5-9996-4a77-9c53-388d4ae437c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:12.548207 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60a0ff5-9996-4a77-9c53-388d4ae437c0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:12.548480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60a0ff5-9996-4a77-9c53-388d4ae437c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:12.548749 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60a0ff5-9996-4a77-9c53-388d4ae437c0 None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:12.549890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60a0ff5-9996-4a77-9c53-388d4ae437c0 None admin] RBAC: Authorizing `identity:get_project(project_id=16a6348ce8d34348b34d853f76445b3a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:12.551377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60a0ff5-9996-4a77-9c53-388d4ae437c0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:12.552596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7ff8fb-eb83-4dfb-bcb3-5e85900a4195 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:12.553169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 132/263] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:12 2022] GET /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a => generated 302 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:12.576635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60a0ff5-9996-4a77-9c53-388d4ae437c0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:12.595824 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21992023-be5e-4339-b428-9f96ffed15bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:12.596436 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21992023-be5e-4339-b428-9f96ffed15bb None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:12.596713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21992023-be5e-4339-b428-9f96ffed15bb None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:12.596982 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21992023-be5e-4339-b428-9f96ffed15bb None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:12.597647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21992023-be5e-4339-b428-9f96ffed15bb None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:12.598008 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21992023-be5e-4339-b428-9f96ffed15bb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:12.599455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21992023-be5e-4339-b428-9f96ffed15bb None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:12.608314 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 132/264] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:12 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:12.615815 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21992023-be5e-4339-b428-9f96ffed15bb None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:12.676864 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:12.700811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:12.753329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60a0ff5-9996-4a77-9c53-388d4ae437c0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:12.798117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f8ba64-affa-4792-91c7-90e4d2320521 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:12.799322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f8ba64-affa-4792-91c7-90e4d2320521 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:12.800275 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f8ba64-affa-4792-91c7-90e4d2320521 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:12.801031 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f8ba64-affa-4792-91c7-90e4d2320521 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:12.801965 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f8ba64-affa-4792-91c7-90e4d2320521 None admin] RBAC: Adding query filter params (role.id=66a36c858aab40688080eed6ceace3a0, scope.project.id=16a6348ce8d34348b34d853f76445b3a, user.id=33fab7c55b3a46b5aa91e225ea277804) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:12.802511 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f8ba64-affa-4792-91c7-90e4d2320521 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:12.802869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21992023-be5e-4339-b428-9f96ffed15bb None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:12.804745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f8ba64-affa-4792-91c7-90e4d2320521 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:12.819005 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 133/265] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:05:12 2022] GET /identity/v3/role_assignments?user.id=33fab7c55b3a46b5aa91e225ea277804&scope.project.id=16a6348ce8d34348b34d853f76445b3a&role.id=66a36c858aab40688080eed6ceace3a0 => generated 608 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:12.851758 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb1e6a6-2989-4493-a6f7-dd716f7eac14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:12.853299 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb1e6a6-2989-4493-a6f7-dd716f7eac14 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:12.853801 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb1e6a6-2989-4493-a6f7-dd716f7eac14 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:12.854268 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb1e6a6-2989-4493-a6f7-dd716f7eac14 None admin] PATH_INFO: `/v3/users/33fab7c55b3a46b5aa91e225ea277804` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:12.855724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb1e6a6-2989-4493-a6f7-dd716f7eac14 None admin] RBAC: Authorizing `identity:get_user(user_id=33fab7c55b3a46b5aa91e225ea277804)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:12.857417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb1e6a6-2989-4493-a6f7-dd716f7eac14 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:12.859439 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 133/266] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:12 2022] GET /identity/v3/users/33fab7c55b3a46b5aa91e225ea277804 => generated 283 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:12.865041 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f8ba64-affa-4792-91c7-90e4d2320521 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:12.989148 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f8ba64-affa-4792-91c7-90e4d2320521 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:13.021850 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:13.028764 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c3dc7c-0567-438c-9a94-bfda188a3d5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:13.029391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c3dc7c-0567-438c-9a94-bfda188a3d5c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:13.029671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c3dc7c-0567-438c-9a94-bfda188a3d5c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:13.029942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c3dc7c-0567-438c-9a94-bfda188a3d5c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:13.106952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c3dc7c-0567-438c-9a94-bfda188a3d5c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:13.108367 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c3dc7c-0567-438c-9a94-bfda188a3d5c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:13.219540 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 134/267] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:13.224912 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb1e6a6-2989-4493-a6f7-dd716f7eac14 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:13.347998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb1e6a6-2989-4493-a6f7-dd716f7eac14 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:13.350318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:13.391319 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c88b2d-ba26-4987-a5d7-a810d604bd76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:13.392418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c88b2d-ba26-4987-a5d7-a810d604bd76 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:13.392898 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c88b2d-ba26-4987-a5d7-a810d604bd76 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:13.393386 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c88b2d-ba26-4987-a5d7-a810d604bd76 None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:13.394876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c88b2d-ba26-4987-a5d7-a810d604bd76 None admin] RBAC: Authorizing `identity:get_project(project_id=16a6348ce8d34348b34d853f76445b3a)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:13.396650 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c88b2d-ba26-4987-a5d7-a810d604bd76 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:13.399568 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 134/268] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:13 2022] GET /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a => generated 302 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:13.416540 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c3dc7c-0567-438c-9a94-bfda188a3d5c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:13.465643 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:13.526313 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebceedd1-6d4a-492a-80ac-4f4b482646fe None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:13.526592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebceedd1-6d4a-492a-80ac-4f4b482646fe None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:13.530897 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebceedd1-6d4a-492a-80ac-4f4b482646fe None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:13.530897 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 135/269] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:13.534306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f40dfc-85cc-41e2-989c-51301dffc489 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:13.534582 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f40dfc-85cc-41e2-989c-51301dffc489 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:13.534880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f40dfc-85cc-41e2-989c-51301dffc489 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:13.556038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-88f40dfc-85cc-41e2-989c-51301dffc489 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:13.557572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-88f40dfc-85cc-41e2-989c-51301dffc489 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:13.583428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c3dc7c-0567-438c-9a94-bfda188a3d5c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:13.650473 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d16980-4057-4ee9-b3dd-454ba1452380 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:13.651868 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d16980-4057-4ee9-b3dd-454ba1452380 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:13.652148 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d16980-4057-4ee9-b3dd-454ba1452380 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:13.652420 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d16980-4057-4ee9-b3dd-454ba1452380 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:13.653147 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d16980-4057-4ee9-b3dd-454ba1452380 None admin] RBAC: Adding query filter params (role.id=cd616e816a394068867091116a3f1ce0, scope.project.id=16a6348ce8d34348b34d853f76445b3a, user.id=33fab7c55b3a46b5aa91e225ea277804) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:13.653457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d16980-4057-4ee9-b3dd-454ba1452380 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:13.654961 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d16980-4057-4ee9-b3dd-454ba1452380 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:13.665993 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 135/270] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:05:13 2022] GET /identity/v3/role_assignments?user.id=33fab7c55b3a46b5aa91e225ea277804&scope.project.id=16a6348ce8d34348b34d853f76445b3a&role.id=cd616e816a394068867091116a3f1ce0 => generated 608 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:13.683804 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 136/271] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 01:05:13.706376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0291e0ac-54e2-4bad-858c-7ed014b7d2e1 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:13.707103 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0291e0ac-54e2-4bad-858c-7ed014b7d2e1 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:13.707588 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0291e0ac-54e2-4bad-858c-7ed014b7d2e1 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:13.709218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 136/272] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:13.720944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43420b8e-c170-4655-b83d-f6a622ad3be2 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:13.721289 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43420b8e-c170-4655-b83d-f6a622ad3be2 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:13.721493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43420b8e-c170-4655-b83d-f6a622ad3be2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:13.752139 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-43420b8e-c170-4655-b83d-f6a622ad3be2 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:13.753391 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-43420b8e-c170-4655-b83d-f6a622ad3be2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:13.836660 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 137/273] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:14.007369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0291e0ac-54e2-4bad-858c-7ed014b7d2e1 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:14.112777 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:14.125337 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0291e0ac-54e2-4bad-858c-7ed014b7d2e1 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:14.157674 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43420b8e-c170-4655-b83d-f6a622ad3be2 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:14.166164 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2339b2-6282-4465-8342-b287a64fdd4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:14.166798 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2339b2-6282-4465-8342-b287a64fdd4b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:14.167103 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2339b2-6282-4465-8342-b287a64fdd4b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:14.167417 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2339b2-6282-4465-8342-b287a64fdd4b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:14.168971 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e2339b2-6282-4465-8342-b287a64fdd4b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:14.170398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e2339b2-6282-4465-8342-b287a64fdd4b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:14.171990 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 137/274] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:14.177307 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2339b2-6282-4465-8342-b287a64fdd4b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:14.202417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:14.281367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43420b8e-c170-4655-b83d-f6a622ad3be2 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:14.291098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2339b2-6282-4465-8342-b287a64fdd4b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:14.317043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ebabbf-ea4c-4ac1-b133-6f5917d8cf6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:14.317666 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ebabbf-ea4c-4ac1-b133-6f5917d8cf6a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:14.317944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ebabbf-ea4c-4ac1-b133-6f5917d8cf6a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:14.318215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ebabbf-ea4c-4ac1-b133-6f5917d8cf6a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:14.326493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84354863-d9a8-4c78-8cdb-6457b8fc6c46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:14.327127 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84354863-d9a8-4c78-8cdb-6457b8fc6c46 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:14.327400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84354863-d9a8-4c78-8cdb-6457b8fc6c46 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:14.327664 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84354863-d9a8-4c78-8cdb-6457b8fc6c46 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:14.328302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84354863-d9a8-4c78-8cdb-6457b8fc6c46 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:14.329032 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ebabbf-ea4c-4ac1-b133-6f5917d8cf6a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:14.329725 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84354863-d9a8-4c78-8cdb-6457b8fc6c46 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:14.333814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-f7ebabbf-ea4c-4ac1-b133-6f5917d8cf6a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:14.333814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:14.333814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:14.333814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:14.333814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:14.333814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:14.333814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:14.333814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:14.333814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:14.333814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:14.333814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:14.333814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:14.333814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:14.333814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:14.333814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:14.333814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:14.334647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:14.334647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:14.334647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:14.334647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:14.334647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:14.334647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:14.334647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:14.334647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:14.334647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:14.334647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:14.334647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:14.334647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:14.334647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:14.334647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:14.334647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:14.334647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:14.335978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:14.335978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:14.335978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:14.335978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:14.335978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:14.335978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:14.335978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:14.335978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:14.335978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:14.335978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:14.335978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:14.335978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:14.335978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 138/275] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:14 2022] GET /identity/v3/roles/member => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:14.336671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2269445c-54d5-4c68-85b9-80b92b70d841 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:14.336671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2269445c-54d5-4c68-85b9-80b92b70d841 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:14.337150 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2269445c-54d5-4c68-85b9-80b92b70d841 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:14.338903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 139/276] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:14.340739 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2269445c-54d5-4c68-85b9-80b92b70d841 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:14.359492 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 138/277] 10.222.0.15 () {60 vars in 1132 bytes} [Tue Sep 13 01:05:14 2022] POST /identity/v3/users => generated 293 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:05:14.364881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d90871-d961-434b-bc29-b041325c5ec7 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:14.365545 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d90871-d961-434b-bc29-b041325c5ec7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:14.366035 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d90871-d961-434b-bc29-b041325c5ec7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:14.394511 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-32d90871-d961-434b-bc29-b041325c5ec7 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:14.396913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-32d90871-d961-434b-bc29-b041325c5ec7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:14.425501 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:14.466803 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:14.482180 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2269445c-54d5-4c68-85b9-80b92b70d841 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:14.513315 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 139/278] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:14.528236 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7288683-3607-4cf7-8aba-c92c42bf4863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:14.529074 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7288683-3607-4cf7-8aba-c92c42bf4863 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:14.529558 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7288683-3607-4cf7-8aba-c92c42bf4863 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:14.529930 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7288683-3607-4cf7-8aba-c92c42bf4863 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:14.530703 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7288683-3607-4cf7-8aba-c92c42bf4863 None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:14.531153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7288683-3607-4cf7-8aba-c92c42bf4863 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:14.533201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7288683-3607-4cf7-8aba-c92c42bf4863 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:14.542205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 140/279] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:14 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:14.548239 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d90871-d961-434b-bc29-b041325c5ec7 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:14.660148 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d90871-d961-434b-bc29-b041325c5ec7 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:14.698762 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302df8a4-67dc-403e-b4f9-b72c8d110e6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:14.699489 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302df8a4-67dc-403e-b4f9-b72c8d110e6e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:14.699735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302df8a4-67dc-403e-b4f9-b72c8d110e6e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:14.699973 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302df8a4-67dc-403e-b4f9-b72c8d110e6e None admin] PATH_INFO: `/v3/users/460b15fbb63f494192dfdc0bcc1a60cf` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:14.714776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-302df8a4-67dc-403e-b4f9-b72c8d110e6e None admin] RBAC: Authorizing `identity:get_user(user_id=460b15fbb63f494192dfdc0bcc1a60cf)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:14.716363 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-302df8a4-67dc-403e-b4f9-b72c8d110e6e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:14.718261 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 140/280] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:14 2022] GET /identity/v3/users/460b15fbb63f494192dfdc0bcc1a60cf => generated 293 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:14.724208 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7288683-3607-4cf7-8aba-c92c42bf4863 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:14.760083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff7df04-2745-493e-92b8-8b12eda215c2 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:14.760363 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff7df04-2745-493e-92b8-8b12eda215c2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:14.760636 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff7df04-2745-493e-92b8-8b12eda215c2 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:14.761974 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 141/281] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:05:14.767452 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce0d4f3-b58e-4637-b403-9cbe052dcd5a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:14.767720 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce0d4f3-b58e-4637-b403-9cbe052dcd5a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:14.767984 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce0d4f3-b58e-4637-b403-9cbe052dcd5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:14.784648 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:14.791108 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-0ce0d4f3-b58e-4637-b403-9cbe052dcd5a None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:14.792567 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0ce0d4f3-b58e-4637-b403-9cbe052dcd5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:14.859894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7288683-3607-4cf7-8aba-c92c42bf4863 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:14.875910 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 142/282] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 01:05:14.877066 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce0d4f3-b58e-4637-b403-9cbe052dcd5a None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:14.923963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e90f26-8a26-4a66-81d3-0adfd6645bd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:14.925647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e90f26-8a26-4a66-81d3-0adfd6645bd6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:14.926880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e90f26-8a26-4a66-81d3-0adfd6645bd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:14.927353 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e90f26-8a26-4a66-81d3-0adfd6645bd6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:14.929454 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e90f26-8a26-4a66-81d3-0adfd6645bd6 None admin] RBAC: Adding query filter params (role.id=6f71034643cb4e8caee299f5d0bb863f, user.id=460b15fbb63f494192dfdc0bcc1a60cf, scope.system=all) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:14.929961 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e90f26-8a26-4a66-81d3-0adfd6645bd6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:14.932342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e90f26-8a26-4a66-81d3-0adfd6645bd6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:14.943448 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 141/283] 10.222.0.15 () {56 vars in 1308 bytes} [Tue Sep 13 01:05:14 2022] GET /identity/v3/role_assignments?user.id=460b15fbb63f494192dfdc0bcc1a60cf&scope.system=all&role.id=6f71034643cb4e8caee299f5d0bb863f => generated 226 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:15.025807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce0d4f3-b58e-4637-b403-9cbe052dcd5a None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:15.081041 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5eb7e3-e51f-4714-9ac7-7f88ff0f6a96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:15.081994 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5eb7e3-e51f-4714-9ac7-7f88ff0f6a96 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:15.082491 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5eb7e3-e51f-4714-9ac7-7f88ff0f6a96 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:15.082999 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5eb7e3-e51f-4714-9ac7-7f88ff0f6a96 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:15.092267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:15.101326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad5eb7e3-e51f-4714-9ac7-7f88ff0f6a96 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:15.107836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-ad5eb7e3-e51f-4714-9ac7-7f88ff0f6a96 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 01:05:15.107836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:15.107836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:15.107836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:15.107836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:15.107836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:15.107836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:15.107836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:15.107836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:15.107836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:15.107836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:15.107836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:15.107836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:15.107836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:15.107836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:15.107836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:15.108710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:15.108710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:15.108710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:15.108710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:15.108710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:15.108710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:15.108710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:15.108710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:15.108710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:15.108710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:15.108710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:15.108710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:15.108710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:15.108710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:15.108710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:15.108710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:15.109483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:15.109483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:15.109483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:15.109483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:15.109483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:15.109483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:15.109483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:15.109483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:15.109483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:15.109483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:15.109483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 01:05:15.109483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:15.111770 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 143/284] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:14 2022] GET /identity/v3/roles/reader => generated 84 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:15.117588 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e90f26-8a26-4a66-81d3-0adfd6645bd6 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:15.147894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:15.248988 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e90f26-8a26-4a66-81d3-0adfd6645bd6 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:15.295588 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8397c2fa-1303-4af2-b99f-58d335b58e1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:15.296212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8397c2fa-1303-4af2-b99f-58d335b58e1b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:15.296495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8397c2fa-1303-4af2-b99f-58d335b58e1b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:15.296768 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8397c2fa-1303-4af2-b99f-58d335b58e1b None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:15.297436 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8397c2fa-1303-4af2-b99f-58d335b58e1b None admin] RBAC: Adding query filter params (name=reader) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:15.297739 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8397c2fa-1303-4af2-b99f-58d335b58e1b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:15.299154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8397c2fa-1303-4af2-b99f-58d335b58e1b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:15.304830 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5eb7e3-e51f-4714-9ac7-7f88ff0f6a96 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:15.310008 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 142/285] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:15 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:15.314837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8397c2fa-1303-4af2-b99f-58d335b58e1b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:15.446590 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:15.467729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5eb7e3-e51f-4714-9ac7-7f88ff0f6a96 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:15.490816 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8397c2fa-1303-4af2-b99f-58d335b58e1b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:15.523872 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9804418e-3edb-4aa6-a913-22d55f54680d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:15.525051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9804418e-3edb-4aa6-a913-22d55f54680d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:15.525324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9804418e-3edb-4aa6-a913-22d55f54680d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:15.525587 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9804418e-3edb-4aa6-a913-22d55f54680d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:15.547224 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9804418e-3edb-4aa6-a913-22d55f54680d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:15.556381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-9804418e-3edb-4aa6-a913-22d55f54680d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:15.556381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:15.556381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:15.556381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:15.556381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:15.556381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:15.556381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:15.556381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:15.556381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:15.556381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:15.556381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:15.556381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:15.556381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:15.556381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:15.556381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:15.556381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:15.557241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:15.557241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:15.557241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:15.557241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:15.557241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:15.557241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:15.557241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:15.557241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:15.557241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:15.557241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:15.557241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:15.557241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:15.557241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:15.557241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:15.557241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:15.557241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:15.558010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:15.558010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:15.558010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:15.558010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:15.558010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:15.558010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:15.558010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:15.558010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:15.558010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:15.558010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:15.558010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:15.558010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:15.558010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 144/286] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:05:15.565151 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c64e98-3934-47b6-b8de-86ea3c2ea68e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:15.565790 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c64e98-3934-47b6-b8de-86ea3c2ea68e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:15.566067 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c64e98-3934-47b6-b8de-86ea3c2ea68e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:15.566333 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c64e98-3934-47b6-b8de-86ea3c2ea68e None admin] PATH_INFO: `/v3/users/b028bffe18854cc9b2980b8c005aac4d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:15.567472 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c64e98-3934-47b6-b8de-86ea3c2ea68e None admin] RBAC: Authorizing `identity:get_user(user_id=b028bffe18854cc9b2980b8c005aac4d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:15.568967 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c64e98-3934-47b6-b8de-86ea3c2ea68e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:15.570520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 143/287] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:15 2022] GET /identity/v3/users/b028bffe18854cc9b2980b8c005aac4d => generated 297 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:15.576040 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9804418e-3edb-4aa6-a913-22d55f54680d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:15.585723 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c64e98-3934-47b6-b8de-86ea3c2ea68e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:15.659482 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:15.742637 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9804418e-3edb-4aa6-a913-22d55f54680d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:15.782360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d482c0f-25b1-454b-ae70-e264fe4f209e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:15.783045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d482c0f-25b1-454b-ae70-e264fe4f209e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:15.784482 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d482c0f-25b1-454b-ae70-e264fe4f209e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:15.784768 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d482c0f-25b1-454b-ae70-e264fe4f209e None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:15.787034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d482c0f-25b1-454b-ae70-e264fe4f209e None admin] RBAC: Authorizing `identity:get_project(project_id=16a6348ce8d34348b34d853f76445b3a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:15.788462 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d482c0f-25b1-454b-ae70-e264fe4f209e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:15.792286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 145/288] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:15 2022] GET /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a => generated 302 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:15.795975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d482c0f-25b1-454b-ae70-e264fe4f209e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:15.802804 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c64e98-3934-47b6-b8de-86ea3c2ea68e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:15.870480 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a07456-00e9-4495-80c0-c164d1e2f7b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:15.871128 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a07456-00e9-4495-80c0-c164d1e2f7b9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:15.871404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a07456-00e9-4495-80c0-c164d1e2f7b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:15.871671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a07456-00e9-4495-80c0-c164d1e2f7b9 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:15.872331 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a07456-00e9-4495-80c0-c164d1e2f7b9 None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:15.872630 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a07456-00e9-4495-80c0-c164d1e2f7b9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:15.873998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a07456-00e9-4495-80c0-c164d1e2f7b9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:15.891136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 144/289] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:15.895766 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a07456-00e9-4495-80c0-c164d1e2f7b9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:15.911421 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:15.985699 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d482c0f-25b1-454b-ae70-e264fe4f209e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:16.031298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:16.050280 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa9ddc0-a8fe-4b51-9e29-506cc3058a84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:16.050976 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa9ddc0-a8fe-4b51-9e29-506cc3058a84 None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:16.051259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa9ddc0-a8fe-4b51-9e29-506cc3058a84 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:16.051533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa9ddc0-a8fe-4b51-9e29-506cc3058a84 None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a/users/b028bffe18854cc9b2980b8c005aac4d/roles/6f98018cf28149aa84201e6dc4a43c2d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:16.056024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa9ddc0-a8fe-4b51-9e29-506cc3058a84 None admin] RBAC: Authorizing `identity:create_grant(project_id=16a6348ce8d34348b34d853f76445b3a, user_id=b028bffe18854cc9b2980b8c005aac4d, role_id=6f98018cf28149aa84201e6dc4a43c2d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:16.057466 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa9ddc0-a8fe-4b51-9e29-506cc3058a84 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:16.082611 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 146/290] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:05:15 2022] PUT /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a/users/b028bffe18854cc9b2980b8c005aac4d/roles/6f98018cf28149aa84201e6dc4a43c2d => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:05:16.085205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a07456-00e9-4495-80c0-c164d1e2f7b9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:16.129675 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e611d4e7-4b19-4f17-b84b-20883b2fc6a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:16.130414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e611d4e7-4b19-4f17-b84b-20883b2fc6a8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:16.130694 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e611d4e7-4b19-4f17-b84b-20883b2fc6a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:16.131044 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e611d4e7-4b19-4f17-b84b-20883b2fc6a8 None admin] PATH_INFO: `/v3/users/6a22d9a5c1634cdb81a5c8e05b18a550` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:16.132334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e611d4e7-4b19-4f17-b84b-20883b2fc6a8 None admin] RBAC: Authorizing `identity:get_user(user_id=6a22d9a5c1634cdb81a5c8e05b18a550)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:16.134385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e611d4e7-4b19-4f17-b84b-20883b2fc6a8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:16.136637 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 145/291] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:15 2022] GET /identity/v3/users/6a22d9a5c1634cdb81a5c8e05b18a550 => generated 297 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:16.142731 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa9ddc0-a8fe-4b51-9e29-506cc3058a84 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:16.278626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa9ddc0-a8fe-4b51-9e29-506cc3058a84 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:16.314491 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0dfa06-881a-48e7-ad93-6cbab19104ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:16.315129 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0dfa06-881a-48e7-ad93-6cbab19104ba None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:16.315393 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0dfa06-881a-48e7-ad93-6cbab19104ba None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:16.315713 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0dfa06-881a-48e7-ad93-6cbab19104ba None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:16.371596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:16.390532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0dfa06-881a-48e7-ad93-6cbab19104ba None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:16.392011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0dfa06-881a-48e7-ad93-6cbab19104ba None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:16.505420 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 147/292] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:16.516241 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e611d4e7-4b19-4f17-b84b-20883b2fc6a8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:16.623332 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:16.648596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e611d4e7-4b19-4f17-b84b-20883b2fc6a8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:16.695403 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfff914-0832-46a2-bc4f-4a87529386ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:16.696064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfff914-0832-46a2-bc4f-4a87529386ea None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:16.696346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfff914-0832-46a2-bc4f-4a87529386ea None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:16.696616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfff914-0832-46a2-bc4f-4a87529386ea None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:16.697745 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfff914-0832-46a2-bc4f-4a87529386ea None admin] RBAC: Authorizing `identity:get_project(project_id=16a6348ce8d34348b34d853f76445b3a)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:16.699272 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfff914-0832-46a2-bc4f-4a87529386ea None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:16.706185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 146/293] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:16 2022] GET /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a => generated 302 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:16.709427 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0dfa06-881a-48e7-ad93-6cbab19104ba None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:16.818027 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:16.837122 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0dfa06-881a-48e7-ad93-6cbab19104ba None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:16.874226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1eab3b4-cce8-4d4e-9f03-87b8d8f28888 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:16.874758 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1eab3b4-cce8-4d4e-9f03-87b8d8f28888 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:16.875073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1eab3b4-cce8-4d4e-9f03-87b8d8f28888 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:16.875480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1eab3b4-cce8-4d4e-9f03-87b8d8f28888 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:16.876217 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1eab3b4-cce8-4d4e-9f03-87b8d8f28888 None admin] RBAC: Adding query filter params (role.id=6f71034643cb4e8caee299f5d0bb863f, scope.project.id=16a6348ce8d34348b34d853f76445b3a, user.id=6a22d9a5c1634cdb81a5c8e05b18a550) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:16.876521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1eab3b4-cce8-4d4e-9f03-87b8d8f28888 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:16.877943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1eab3b4-cce8-4d4e-9f03-87b8d8f28888 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:16.884546 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 148/294] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:05:16 2022] GET /identity/v3/role_assignments?user.id=6a22d9a5c1634cdb81a5c8e05b18a550&scope.project.id=16a6348ce8d34348b34d853f76445b3a&role.id=6f71034643cb4e8caee299f5d0bb863f => generated 608 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:17.077294 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d647e8-6016-4f29-97c6-6c5d2dc67f56 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:17.077688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d647e8-6016-4f29-97c6-6c5d2dc67f56 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:17.077864 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d647e8-6016-4f29-97c6-6c5d2dc67f56 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:17.079250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 147/295] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:17.084661 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dde5fba-1a39-4c64-b2c0-cf2ad4922052 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:17.084949 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dde5fba-1a39-4c64-b2c0-cf2ad4922052 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:17.085222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dde5fba-1a39-4c64-b2c0-cf2ad4922052 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:17.107259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8dde5fba-1a39-4c64-b2c0-cf2ad4922052 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:17.108456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8dde5fba-1a39-4c64-b2c0-cf2ad4922052 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:17.191487 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 149/296] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:17.252428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfffb3b3-6caf-4c46-bf75-f865dd246ddc None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:17.253053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfffb3b3-6caf-4c46-bf75-f865dd246ddc None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:17.253521 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfffb3b3-6caf-4c46-bf75-f865dd246ddc None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:17.255410 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 148/297] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:05:17.260964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f617b10-4a9e-43ba-8e8d-9a6475f0867c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:17.261248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f617b10-4a9e-43ba-8e8d-9a6475f0867c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:17.261524 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f617b10-4a9e-43ba-8e8d-9a6475f0867c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:17.283735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7f617b10-4a9e-43ba-8e8d-9a6475f0867c None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:17.284952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7f617b10-4a9e-43ba-8e8d-9a6475f0867c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:17.369415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 150/298] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:17.476068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfffb3b3-6caf-4c46-bf75-f865dd246ddc None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:17.583371 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfffb3b3-6caf-4c46-bf75-f865dd246ddc None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:17.590669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d10996-5f10-46b3-9218-499053cc9106 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:17.590984 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d10996-5f10-46b3-9218-499053cc9106 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:17.591259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d10996-5f10-46b3-9218-499053cc9106 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:17.592811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 151/299] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:17.597622 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d4fac1-7d9a-4e1a-adc5-782c1972b9b1 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:17.597904 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d4fac1-7d9a-4e1a-adc5-782c1972b9b1 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:17.598172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d4fac1-7d9a-4e1a-adc5-782c1972b9b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:17.616751 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e2d4fac1-7d9a-4e1a-adc5-782c1972b9b1 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:17.617976 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e2d4fac1-7d9a-4e1a-adc5-782c1972b9b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:17.623131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c67efcc-42b5-4b99-9375-db3e7aff2579 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:17.623820 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c67efcc-42b5-4b99-9375-db3e7aff2579 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:17.624099 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c67efcc-42b5-4b99-9375-db3e7aff2579 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:17.624373 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c67efcc-42b5-4b99-9375-db3e7aff2579 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:17.627174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:17.639034 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c67efcc-42b5-4b99-9375-db3e7aff2579 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:17.646690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-5c67efcc-42b5-4b99-9375-db3e7aff2579 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 01:05:17.646690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:17.646690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:17.646690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:17.646690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:17.646690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:17.646690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:17.646690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:17.646690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:17.646690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:17.646690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:17.646690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:17.646690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:17.646690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:17.646690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:17.646690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:17.647578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:17.647578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:17.647578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:17.647578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:17.647578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:17.647578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:17.647578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:17.647578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:17.647578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:17.647578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:17.647578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:17.647578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:17.647578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:17.647578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:17.647578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:17.647578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:17.648358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:17.648358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:17.648358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:17.648358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:17.648358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:17.648358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:17.648358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:17.648358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:17.648358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:17.648358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:17.648358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 01:05:17.648358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:17.650941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 149/300] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:17 2022] GET /identity/v3/roles/reader => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:17.654377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c67efcc-42b5-4b99-9375-db3e7aff2579 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:17.659141 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:17.694255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 152/301] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 01:05:17.695806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d4fac1-7d9a-4e1a-adc5-782c1972b9b1 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:17.761644 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c67efcc-42b5-4b99-9375-db3e7aff2579 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:17.796470 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e4caf7-e5bc-4e9e-a2a5-c4dc8ae092be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:17.797076 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e4caf7-e5bc-4e9e-a2a5-c4dc8ae092be None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:17.797355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e4caf7-e5bc-4e9e-a2a5-c4dc8ae092be None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:17.797629 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e4caf7-e5bc-4e9e-a2a5-c4dc8ae092be None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:17.798287 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e4caf7-e5bc-4e9e-a2a5-c4dc8ae092be None admin] RBAC: Adding query filter params (name=reader) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:17.798590 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e4caf7-e5bc-4e9e-a2a5-c4dc8ae092be None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:17.799994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e4caf7-e5bc-4e9e-a2a5-c4dc8ae092be None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:17.815647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 150/302] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:17 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:17.816853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e4caf7-e5bc-4e9e-a2a5-c4dc8ae092be None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:17.823579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d4fac1-7d9a-4e1a-adc5-782c1972b9b1 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:17.871578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9284659-6b7a-4a2b-bf70-f270b658e3da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:17.872218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9284659-6b7a-4a2b-bf70-f270b658e3da None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:17.872494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9284659-6b7a-4a2b-bf70-f270b658e3da None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:17.872763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9284659-6b7a-4a2b-bf70-f270b658e3da None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:17.874728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9284659-6b7a-4a2b-bf70-f270b658e3da None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:17.877580 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9284659-6b7a-4a2b-bf70-f270b658e3da None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:17.882887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 153/303] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:17 2022] GET /identity/v3/domains/default => generated 199 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:17.885546 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9284659-6b7a-4a2b-bf70-f270b658e3da None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:17.900488 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:17.903488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:17.940350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e4caf7-e5bc-4e9e-a2a5-c4dc8ae092be None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:17.975150 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad481298-b4a3-4644-8dd7-4eae5d63525f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:17.975776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad481298-b4a3-4644-8dd7-4eae5d63525f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:17.976054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad481298-b4a3-4644-8dd7-4eae5d63525f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:17.976327 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad481298-b4a3-4644-8dd7-4eae5d63525f None admin] PATH_INFO: `/v3/users/7b88d371cd594f4496ca02b449221e6f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:17.993186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad481298-b4a3-4644-8dd7-4eae5d63525f None admin] RBAC: Authorizing `identity:get_user(user_id=7b88d371cd594f4496ca02b449221e6f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:17.994671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad481298-b4a3-4644-8dd7-4eae5d63525f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:17.996491 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 151/304] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:17 2022] GET /identity/v3/users/7b88d371cd594f4496ca02b449221e6f => generated 293 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:18.001851 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad481298-b4a3-4644-8dd7-4eae5d63525f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:18.010188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9284659-6b7a-4a2b-bf70-f270b658e3da None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:18.045907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b134d4b9-c9e3-4290-a20b-4a5f5fa6d133 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:18.046447 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b134d4b9-c9e3-4290-a20b-4a5f5fa6d133 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:18.046724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b134d4b9-c9e3-4290-a20b-4a5f5fa6d133 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:18.047024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b134d4b9-c9e3-4290-a20b-4a5f5fa6d133 None admin] PATH_INFO: `/v3/groups` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:18.047704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b134d4b9-c9e3-4290-a20b-4a5f5fa6d133 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:18.049117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b134d4b9-c9e3-4290-a20b-4a5f5fa6d133 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:18.074889 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 154/305] 10.222.0.15 () {60 vars in 1134 bytes} [Tue Sep 13 01:05:17 2022] POST /identity/v3/groups => generated 229 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:05:18.074889 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b134d4b9-c9e3-4290-a20b-4a5f5fa6d133 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:18.122263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad481298-b4a3-4644-8dd7-4eae5d63525f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:18.148108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:18.157674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:18.165313 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1050e9b-82c8-4ad2-b466-e80c2227c234 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:18.166409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1050e9b-82c8-4ad2-b466-e80c2227c234 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:18.166955 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1050e9b-82c8-4ad2-b466-e80c2227c234 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:18.167448 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1050e9b-82c8-4ad2-b466-e80c2227c234 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:18.185696 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1050e9b-82c8-4ad2-b466-e80c2227c234 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:18.192005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-d1050e9b-82c8-4ad2-b466-e80c2227c234 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:18.192005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:18.192005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:18.192005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:18.192005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:18.192005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:18.192005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:18.192005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:18.192005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:18.192005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:18.192005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:18.192005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:18.192005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:18.192005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:18.192005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:18.192005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:18.192977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:18.192977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:18.192977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:18.192977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:18.192977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:18.192977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:18.192977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:18.192977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:18.192977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:18.192977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:18.192977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:18.192977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:18.192977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:18.192977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:18.192977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:18.192977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:18.193772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:18.193772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:18.193772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:18.193772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:18.193772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:18.193772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:18.193772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:18.193772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:18.193772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:18.193772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:18.193772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:18.193772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:18.193772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 152/306] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:18.199996 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1050e9b-82c8-4ad2-b466-e80c2227c234 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:18.212706 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b134d4b9-c9e3-4290-a20b-4a5f5fa6d133 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:18.257769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597dd73a-413c-441b-a6d4-7120ae3c956a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:18.258442 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597dd73a-413c-441b-a6d4-7120ae3c956a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:18.258721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597dd73a-413c-441b-a6d4-7120ae3c956a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:18.259028 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597dd73a-413c-441b-a6d4-7120ae3c956a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:18.259766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597dd73a-413c-441b-a6d4-7120ae3c956a None admin] RBAC: Adding query filter params (role.id=6f98018cf28149aa84201e6dc4a43c2d, user.id=7b88d371cd594f4496ca02b449221e6f, scope.system=all) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:18.263223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597dd73a-413c-441b-a6d4-7120ae3c956a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:18.265031 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597dd73a-413c-441b-a6d4-7120ae3c956a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:18.272885 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 155/307] 10.222.0.15 () {56 vars in 1308 bytes} [Tue Sep 13 01:05:18 2022] GET /identity/v3/role_assignments?user.id=7b88d371cd594f4496ca02b449221e6f&scope.system=all&role.id=6f98018cf28149aa84201e6dc4a43c2d => generated 226 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:18.336111 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1050e9b-82c8-4ad2-b466-e80c2227c234 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:18.381252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec704463-d3e9-43f9-902f-f8614f932460 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:18.382248 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec704463-d3e9-43f9-902f-f8614f932460 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:18.382731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec704463-d3e9-43f9-902f-f8614f932460 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:18.383230 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec704463-d3e9-43f9-902f-f8614f932460 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:18.384683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec704463-d3e9-43f9-902f-f8614f932460 None admin] RBAC: Adding query filter params (name=member) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:18.385194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec704463-d3e9-43f9-902f-f8614f932460 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:18.386812 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec704463-d3e9-43f9-902f-f8614f932460 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:18.397167 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 153/308] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:18.414275 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597dd73a-413c-441b-a6d4-7120ae3c956a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:18.531583 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597dd73a-413c-441b-a6d4-7120ae3c956a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:18.563455 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:18.570441 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b104b98-72de-4fef-adc2-2f0cf0317fe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:18.571450 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b104b98-72de-4fef-adc2-2f0cf0317fe3 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:18.571953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b104b98-72de-4fef-adc2-2f0cf0317fe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:18.572452 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b104b98-72de-4fef-adc2-2f0cf0317fe3 None admin] PATH_INFO: `/v3/users/460b15fbb63f494192dfdc0bcc1a60cf` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:18.573812 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b104b98-72de-4fef-adc2-2f0cf0317fe3 None admin] RBAC: Authorizing `identity:get_user(user_id=460b15fbb63f494192dfdc0bcc1a60cf)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:18.575490 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b104b98-72de-4fef-adc2-2f0cf0317fe3 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:18.577412 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 156/309] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:18 2022] GET /identity/v3/users/460b15fbb63f494192dfdc0bcc1a60cf => generated 293 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:18.583347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec704463-d3e9-43f9-902f-f8614f932460 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:18.632785 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:18.696279 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec704463-d3e9-43f9-902f-f8614f932460 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:18.736854 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91457c9-ece2-4aa1-add3-c34edbe03746 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:18.737850 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91457c9-ece2-4aa1-add3-c34edbe03746 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:18.738328 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91457c9-ece2-4aa1-add3-c34edbe03746 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:18.738823 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91457c9-ece2-4aa1-add3-c34edbe03746 None admin] PATH_INFO: `/v3/system/users/460b15fbb63f494192dfdc0bcc1a60cf/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:18.740451 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91457c9-ece2-4aa1-add3-c34edbe03746 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=460b15fbb63f494192dfdc0bcc1a60cf, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:18.742046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91457c9-ece2-4aa1-add3-c34edbe03746 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:18.754571 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 154/310] 10.222.0.15 () {58 vars in 1334 bytes} [Tue Sep 13 01:05:18 2022] PUT /identity/v3/system/users/460b15fbb63f494192dfdc0bcc1a60cf/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:05:18.787919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c332a34e-c105-47a6-ae58-4b376214672f None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:18.788266 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c332a34e-c105-47a6-ae58-4b376214672f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:18.788465 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c332a34e-c105-47a6-ae58-4b376214672f None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:18.790116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 157/311] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:18.795877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff3946a-7081-4561-952f-ecfe9724afc8 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:18.796176 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff3946a-7081-4561-952f-ecfe9724afc8 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:18.796421 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff3946a-7081-4561-952f-ecfe9724afc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:18.816455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-aff3946a-7081-4561-952f-ecfe9724afc8 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:18.817678 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-aff3946a-7081-4561-952f-ecfe9724afc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:18.893660 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 155/312] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:19.160290 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c332a34e-c105-47a6-ae58-4b376214672f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:19.268826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c332a34e-c105-47a6-ae58-4b376214672f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:19.304118 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c0349e-97a6-4dd4-b090-5810d6319b42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:19.305062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c0349e-97a6-4dd4-b090-5810d6319b42 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:19.306075 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c0349e-97a6-4dd4-b090-5810d6319b42 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:19.306543 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c0349e-97a6-4dd4-b090-5810d6319b42 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:19.317457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c0349e-97a6-4dd4-b090-5810d6319b42 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:19.323076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-85c0349e-97a6-4dd4-b090-5810d6319b42 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 01:05:19.323076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:19.323076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:19.323076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:19.323076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:19.323076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:19.323076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:19.323076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:19.323076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:19.323076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:19.323076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:19.323076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:19.323076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:19.323076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:19.323076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:19.323076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:19.323933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:19.323933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:19.323933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:19.323933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:19.323933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:19.323933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:19.323933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:19.323933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:19.323933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:19.323933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:19.323933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:19.323933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:19.323933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:19.323933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:19.323933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:19.323933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:19.324735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:19.324735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:19.324735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:19.324735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:19.324735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:19.324735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:19.324735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:19.324735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:19.324735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:19.324735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:19.324735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 01:05:19.324735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:19.326922 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 158/313] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:19 2022] GET /identity/v3/roles/reader => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:19.334740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff3946a-7081-4561-952f-ecfe9724afc8 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:19.374988 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:19.443718 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff3946a-7081-4561-952f-ecfe9724afc8 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:19.480103 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc3ff31-af03-468b-a02c-97f3cee5203e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:19.481064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc3ff31-af03-468b-a02c-97f3cee5203e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:19.481560 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc3ff31-af03-468b-a02c-97f3cee5203e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:19.482044 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc3ff31-af03-468b-a02c-97f3cee5203e None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:19.482991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc3ff31-af03-468b-a02c-97f3cee5203e None admin] RBAC: Adding query filter params (name=reader) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:19.483493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc3ff31-af03-468b-a02c-97f3cee5203e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:19.485094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc3ff31-af03-468b-a02c-97f3cee5203e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:19.495063 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 156/314] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:19 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:19.503478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c0349e-97a6-4dd4-b090-5810d6319b42 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:19.566099 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:19.610322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c0349e-97a6-4dd4-b090-5810d6319b42 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:19.644467 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a243bc-0857-4167-aa76-6c745bc12795 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:19.645086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a243bc-0857-4167-aa76-6c745bc12795 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:19.645360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a243bc-0857-4167-aa76-6c745bc12795 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:19.645629 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a243bc-0857-4167-aa76-6c745bc12795 None admin] PATH_INFO: `/v3/users/b028bffe18854cc9b2980b8c005aac4d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:19.646739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a243bc-0857-4167-aa76-6c745bc12795 None admin] RBAC: Authorizing `identity:get_user(user_id=b028bffe18854cc9b2980b8c005aac4d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:19.648182 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a243bc-0857-4167-aa76-6c745bc12795 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:19.650157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 159/315] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:19 2022] GET /identity/v3/users/b028bffe18854cc9b2980b8c005aac4d => generated 297 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:19.656146 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc3ff31-af03-468b-a02c-97f3cee5203e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:19.765337 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc3ff31-af03-468b-a02c-97f3cee5203e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:19.766817 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:19.802631 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ed7d6a-83b9-4d20-b723-2c6125feefc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:19.803273 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ed7d6a-83b9-4d20-b723-2c6125feefc5 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:19.803550 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ed7d6a-83b9-4d20-b723-2c6125feefc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:19.803817 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ed7d6a-83b9-4d20-b723-2c6125feefc5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:19.879427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ed7d6a-83b9-4d20-b723-2c6125feefc5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:19.881318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ed7d6a-83b9-4d20-b723-2c6125feefc5 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:19.984407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 157/316] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:19.992157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a243bc-0857-4167-aa76-6c745bc12795 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:20.097377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a243bc-0857-4167-aa76-6c745bc12795 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:20.127805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:20.134949 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2525955-1754-476e-a3de-0cf9c84574dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:20.136007 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2525955-1754-476e-a3de-0cf9c84574dd None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:20.136678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2525955-1754-476e-a3de-0cf9c84574dd None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:20.136950 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2525955-1754-476e-a3de-0cf9c84574dd None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:20.138127 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2525955-1754-476e-a3de-0cf9c84574dd None admin] RBAC: Authorizing `identity:get_project(project_id=16a6348ce8d34348b34d853f76445b3a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:20.139592 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2525955-1754-476e-a3de-0cf9c84574dd None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:20.141559 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 160/317] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:19 2022] GET /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a => generated 302 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:20.147998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ed7d6a-83b9-4d20-b723-2c6125feefc5 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:20.150743 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:20.253299 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ed7d6a-83b9-4d20-b723-2c6125feefc5 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:20.293136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1405dfa-50b6-4216-bb9a-0ada8b58844e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:20.293761 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1405dfa-50b6-4216-bb9a-0ada8b58844e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:20.294036 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1405dfa-50b6-4216-bb9a-0ada8b58844e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:20.294307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1405dfa-50b6-4216-bb9a-0ada8b58844e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:20.295072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1405dfa-50b6-4216-bb9a-0ada8b58844e None admin] RBAC: Adding query filter params (role.id=6f98018cf28149aa84201e6dc4a43c2d, scope.project.id=16a6348ce8d34348b34d853f76445b3a, user.id=b028bffe18854cc9b2980b8c005aac4d) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:20.295379 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1405dfa-50b6-4216-bb9a-0ada8b58844e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:20.296818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1405dfa-50b6-4216-bb9a-0ada8b58844e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:20.303353 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 158/318] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:05:20 2022] GET /identity/v3/role_assignments?user.id=b028bffe18854cc9b2980b8c005aac4d&scope.project.id=16a6348ce8d34348b34d853f76445b3a&role.id=6f98018cf28149aa84201e6dc4a43c2d => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:20.387257 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7736f9a3-77d3-4ff3-b9f7-d2397ea84a4a None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:20.387547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7736f9a3-77d3-4ff3-b9f7-d2397ea84a4a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:20.387726 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7736f9a3-77d3-4ff3-b9f7-d2397ea84a4a None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:20.389426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 161/319] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:20.395202 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70889d8a-4631-4bf4-8971-6139438c4cf3 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:20.395485 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70889d8a-4631-4bf4-8971-6139438c4cf3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:20.395760 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70889d8a-4631-4bf4-8971-6139438c4cf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:20.425729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-70889d8a-4631-4bf4-8971-6139438c4cf3 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:20.426978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-70889d8a-4631-4bf4-8971-6139438c4cf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:20.503111 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:20.514171 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 159/320] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:20.659446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719a7b60-f401-47d6-b731-29006d5d7d01 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:20.659930 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719a7b60-f401-47d6-b731-29006d5d7d01 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:20.659930 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719a7b60-f401-47d6-b731-29006d5d7d01 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:20.661389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 162/321] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:20.666753 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7790a760-e943-412c-98f5-1339c21fb27e None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:20.667173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7790a760-e943-412c-98f5-1339c21fb27e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:20.667328 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7790a760-e943-412c-98f5-1339c21fb27e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:20.686299 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7790a760-e943-412c-98f5-1339c21fb27e None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:20.687496 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7790a760-e943-412c-98f5-1339c21fb27e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:20.756171 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 160/322] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:20.782609 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719a7b60-f401-47d6-b731-29006d5d7d01 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:20.897276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719a7b60-f401-47d6-b731-29006d5d7d01 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:20.932381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717ffe4e-a7da-491a-9b7b-9242e0876b88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:20.933336 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717ffe4e-a7da-491a-9b7b-9242e0876b88 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:20.933821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717ffe4e-a7da-491a-9b7b-9242e0876b88 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:20.934320 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717ffe4e-a7da-491a-9b7b-9242e0876b88 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:20.935800 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717ffe4e-a7da-491a-9b7b-9242e0876b88 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:20.937496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717ffe4e-a7da-491a-9b7b-9242e0876b88 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:20.940011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 163/323] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:20 2022] GET /identity/v3/domains/default => generated 199 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:20.945407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7790a760-e943-412c-98f5-1339c21fb27e None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:21.030488 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717ffe4e-a7da-491a-9b7b-9242e0876b88 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:21.052216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7790a760-e943-412c-98f5-1339c21fb27e None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:21.079598 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:21.089169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c085eb13-ab34-497b-a200-42a90958119f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:21.089983 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c085eb13-ab34-497b-a200-42a90958119f None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:21.090249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c085eb13-ab34-497b-a200-42a90958119f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:21.090517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c085eb13-ab34-497b-a200-42a90958119f None admin] PATH_INFO: `/v3/groups` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:21.091177 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c085eb13-ab34-497b-a200-42a90958119f None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:21.092602 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c085eb13-ab34-497b-a200-42a90958119f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:21.096861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:21.109998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 161/324] 10.222.0.15 () {60 vars in 1134 bytes} [Tue Sep 13 01:05:20 2022] POST /identity/v3/groups => generated 226 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:05:21.111283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f008f8cd-b6bc-4d31-b3b5-c2359b6acece None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:21.111579 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f008f8cd-b6bc-4d31-b3b5-c2359b6acece None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:21.111855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f008f8cd-b6bc-4d31-b3b5-c2359b6acece None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:21.113298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 162/325] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:21.119971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45e51eb-a770-4058-a929-cb014258c15a None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:21.120303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45e51eb-a770-4058-a929-cb014258c15a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:21.120586 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45e51eb-a770-4058-a929-cb014258c15a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:21.139593 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d45e51eb-a770-4058-a929-cb014258c15a None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:21.140839 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d45e51eb-a770-4058-a929-cb014258c15a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:21.143988 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717ffe4e-a7da-491a-9b7b-9242e0876b88 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:21.184427 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec93584-70e5-4453-8bba-88f4852a5e9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:21.184989 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec93584-70e5-4453-8bba-88f4852a5e9b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:21.185263 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec93584-70e5-4453-8bba-88f4852a5e9b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:21.185604 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec93584-70e5-4453-8bba-88f4852a5e9b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:21.197263 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec93584-70e5-4453-8bba-88f4852a5e9b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:21.203688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-fec93584-70e5-4453-8bba-88f4852a5e9b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 01:05:21.203688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:21.203688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:21.203688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:21.203688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:21.203688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:21.203688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:21.203688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:21.203688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:21.203688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:21.203688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:21.203688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:21.203688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:21.203688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:21.203688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:21.203688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:21.204685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:21.204685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:21.204685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:21.204685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:21.204685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:21.204685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:21.204685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:21.204685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:21.204685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:21.204685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:21.204685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:21.204685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:21.204685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:21.204685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:21.204685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:21.204685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:21.205478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:21.205478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:21.205478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:21.205478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:21.205478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:21.205478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:21.205478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:21.205478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:21.205478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:21.205478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:21.205478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 01:05:21.205478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:21.205478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 164/326] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:21 2022] GET /identity/v3/roles/reader => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:21.210968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec93584-70e5-4453-8bba-88f4852a5e9b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:21.221817 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 163/327] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 01:05:21.320216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec93584-70e5-4453-8bba-88f4852a5e9b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:21.343576 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:21.358600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792cc2e4-664b-4e4e-826d-2fe3d89259a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:21.359317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792cc2e4-664b-4e4e-826d-2fe3d89259a4 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:21.359595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792cc2e4-664b-4e4e-826d-2fe3d89259a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:21.359880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792cc2e4-664b-4e4e-826d-2fe3d89259a4 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:21.360530 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792cc2e4-664b-4e4e-826d-2fe3d89259a4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:21.360832 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792cc2e4-664b-4e4e-826d-2fe3d89259a4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:21.362198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792cc2e4-664b-4e4e-826d-2fe3d89259a4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:21.371124 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 165/328] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:21 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:21.376756 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45e51eb-a770-4058-a929-cb014258c15a None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:21.485911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792cc2e4-664b-4e4e-826d-2fe3d89259a4 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:21.490383 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45e51eb-a770-4058-a929-cb014258c15a None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:21.512688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:21.528859 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f992060a-a699-47b1-8e8b-39074ce952b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:21.529496 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f992060a-a699-47b1-8e8b-39074ce952b0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:21.529773 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f992060a-a699-47b1-8e8b-39074ce952b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:21.530113 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f992060a-a699-47b1-8e8b-39074ce952b0 None admin] PATH_INFO: `/v3/users/7b88d371cd594f4496ca02b449221e6f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:21.531385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f992060a-a699-47b1-8e8b-39074ce952b0 None admin] RBAC: Authorizing `identity:get_user(user_id=7b88d371cd594f4496ca02b449221e6f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:21.533032 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f992060a-a699-47b1-8e8b-39074ce952b0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:21.535180 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 164/329] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:21 2022] GET /identity/v3/users/7b88d371cd594f4496ca02b449221e6f => generated 293 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:21.540469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f992060a-a699-47b1-8e8b-39074ce952b0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:21.616213 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792cc2e4-664b-4e4e-826d-2fe3d89259a4 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:21.662799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0f864d-dbde-42fb-b625-f75e62fd5ab3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:21.663560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0f864d-dbde-42fb-b625-f75e62fd5ab3 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:21.663741 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0f864d-dbde-42fb-b625-f75e62fd5ab3 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:21.664032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0f864d-dbde-42fb-b625-f75e62fd5ab3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:21.669783 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f992060a-a699-47b1-8e8b-39074ce952b0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:21.677161 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0f864d-dbde-42fb-b625-f75e62fd5ab3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:21.684932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-cf0f864d-dbde-42fb-b625-f75e62fd5ab3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:21.684932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:21.684932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:21.684932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:21.684932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:21.684932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:21.684932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:21.684932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:21.684932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:21.684932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:21.684932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:21.684932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:21.684932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:21.684932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:21.684932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:21.684932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:21.685877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:21.685877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:21.685877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:21.685877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:21.685877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:21.685877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:21.685877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:21.685877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:21.685877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:21.685877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:21.685877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:21.685877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:21.685877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:21.685877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:21.685877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:21.685877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:21.686704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:21.686704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:21.686704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:21.686704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:21.686704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:21.686704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:21.686704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:21.686704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:21.686704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:21.686704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:21.686704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:21.686704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:21.686704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 166/330] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:21 2022] GET /identity/v3/roles/member => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:21.692508 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0f864d-dbde-42fb-b625-f75e62fd5ab3 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:21.716172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aadb4131-eba3-413c-a3b5-566af36d9b51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:21.717182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aadb4131-eba3-413c-a3b5-566af36d9b51 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:21.717686 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aadb4131-eba3-413c-a3b5-566af36d9b51 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:21.718231 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aadb4131-eba3-413c-a3b5-566af36d9b51 None admin] PATH_INFO: `/v3/system/users/7b88d371cd594f4496ca02b449221e6f/roles/6f98018cf28149aa84201e6dc4a43c2d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:21.719858 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aadb4131-eba3-413c-a3b5-566af36d9b51 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=7b88d371cd594f4496ca02b449221e6f, role_id=6f98018cf28149aa84201e6dc4a43c2d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:21.721320 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aadb4131-eba3-413c-a3b5-566af36d9b51 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:21.731491 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 165/331] 10.222.0.15 () {58 vars in 1334 bytes} [Tue Sep 13 01:05:21 2022] PUT /identity/v3/system/users/7b88d371cd594f4496ca02b449221e6f/roles/6f98018cf28149aa84201e6dc4a43c2d => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:05:21.760602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:21.823893 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0f864d-dbde-42fb-b625-f75e62fd5ab3 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:21.861695 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cc26d1-62c1-40af-b5d6-a28dded03637 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:21.862321 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cc26d1-62c1-40af-b5d6-a28dded03637 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:21.862595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cc26d1-62c1-40af-b5d6-a28dded03637 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:21.862894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cc26d1-62c1-40af-b5d6-a28dded03637 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:21.863560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96cc26d1-62c1-40af-b5d6-a28dded03637 None admin] RBAC: Adding query filter params (name=member) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:21.863863 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96cc26d1-62c1-40af-b5d6-a28dded03637 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:21.865289 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96cc26d1-62c1-40af-b5d6-a28dded03637 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:21.874801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 167/332] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:21 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:21.880499 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aadb4131-eba3-413c-a3b5-566af36d9b51 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:21.939224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:21.997915 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aadb4131-eba3-413c-a3b5-566af36d9b51 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:22.041134 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e29dea-4178-45b6-a941-121bbc5eda01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:22.041587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e29dea-4178-45b6-a941-121bbc5eda01 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:22.041860 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e29dea-4178-45b6-a941-121bbc5eda01 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:22.042128 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e29dea-4178-45b6-a941-121bbc5eda01 None admin] PATH_INFO: `/v3/users/460b15fbb63f494192dfdc0bcc1a60cf` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:22.044014 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e29dea-4178-45b6-a941-121bbc5eda01 None admin] RBAC: Authorizing `identity:get_user(user_id=460b15fbb63f494192dfdc0bcc1a60cf)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:22.045480 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e29dea-4178-45b6-a941-121bbc5eda01 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:22.047391 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 166/333] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:21 2022] GET /identity/v3/users/460b15fbb63f494192dfdc0bcc1a60cf => generated 293 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:22.053194 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cc26d1-62c1-40af-b5d6-a28dded03637 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:22.164308 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cc26d1-62c1-40af-b5d6-a28dded03637 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:22.171816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:22.200809 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25360ee-e90f-403b-ab09-22593685aa44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:22.201497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25360ee-e90f-403b-ab09-22593685aa44 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:22.201776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25360ee-e90f-403b-ab09-22593685aa44 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:22.202047 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25360ee-e90f-403b-ab09-22593685aa44 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:22.202774 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25360ee-e90f-403b-ab09-22593685aa44 None admin] RBAC: Adding query filter params (role.id=6f71034643cb4e8caee299f5d0bb863f, user.id=460b15fbb63f494192dfdc0bcc1a60cf, scope.system=all) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:22.203114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25360ee-e90f-403b-ab09-22593685aa44 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:22.204657 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25360ee-e90f-403b-ab09-22593685aa44 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:22.219455 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 168/334] 10.222.0.15 () {56 vars in 1308 bytes} [Tue Sep 13 01:05:22 2022] GET /identity/v3/role_assignments?user.id=460b15fbb63f494192dfdc0bcc1a60cf&scope.system=all&role.id=6f71034643cb4e8caee299f5d0bb863f => generated 510 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:24.201091 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338b45e5-7bab-4f38-bf4d-7b2a540a6fc1 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:24.201608 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338b45e5-7bab-4f38-bf4d-7b2a540a6fc1 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:24.201608 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338b45e5-7bab-4f38-bf4d-7b2a540a6fc1 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:24.203265 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 167/335] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:24.204743 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2ee926-e787-4498-a9d1-4227087ac24e None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:24.205123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2ee926-e787-4498-a9d1-4227087ac24e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:24.205294 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2ee926-e787-4498-a9d1-4227087ac24e None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:24.206599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 169/336] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:24.212816 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83914c58-06ce-45b4-b12f-cbe0c90b1ed0 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:24.213100 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83914c58-06ce-45b4-b12f-cbe0c90b1ed0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:24.213378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83914c58-06ce-45b4-b12f-cbe0c90b1ed0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:24.219502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5250eeaa-339a-4208-b7ba-71261ce20a34 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:24.219781 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5250eeaa-339a-4208-b7ba-71261ce20a34 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:24.220052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5250eeaa-339a-4208-b7ba-71261ce20a34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:24.245276 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-83914c58-06ce-45b4-b12f-cbe0c90b1ed0 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:24.246484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-83914c58-06ce-45b4-b12f-cbe0c90b1ed0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:24.265441 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5250eeaa-339a-4208-b7ba-71261ce20a34 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:24.266649 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5250eeaa-339a-4208-b7ba-71261ce20a34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:24.390646 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 168/337] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:24.410816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 170/338] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:24.603859 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9743d636-7313-43ef-bebd-4404dec2d701 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:24.604515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9743d636-7313-43ef-bebd-4404dec2d701 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:24.604992 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9743d636-7313-43ef-bebd-4404dec2d701 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:24.606797 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 169/339] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:24.611988 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cffce90-f5a9-4e97-a6fb-62a3d6d85a94 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:24.612274 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cffce90-f5a9-4e97-a6fb-62a3d6d85a94 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:24.612550 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cffce90-f5a9-4e97-a6fb-62a3d6d85a94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:24.639068 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4cffce90-f5a9-4e97-a6fb-62a3d6d85a94 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:24.640247 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4cffce90-f5a9-4e97-a6fb-62a3d6d85a94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:24.736698 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 171/340] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:24.765769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9743d636-7313-43ef-bebd-4404dec2d701 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:24.843077 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:24.891115 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cffce90-f5a9-4e97-a6fb-62a3d6d85a94 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:24.905543 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9743d636-7313-43ef-bebd-4404dec2d701 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:24.939205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:24.946812 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642c4c14-9d26-4ad8-8164-6bdaf3901dd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:24.947720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642c4c14-9d26-4ad8-8164-6bdaf3901dd4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:24.948137 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642c4c14-9d26-4ad8-8164-6bdaf3901dd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:24.948543 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642c4c14-9d26-4ad8-8164-6bdaf3901dd4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:24.968879 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642c4c14-9d26-4ad8-8164-6bdaf3901dd4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:24.981285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-642c4c14-9d26-4ad8-8164-6bdaf3901dd4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:24.981285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:24.981285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:24.981285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:24.981285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:24.981285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:24.981285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:24.981285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:24.981285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:24.981285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:24.981285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:24.981285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:24.981285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:24.981285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:24.981285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:24.981285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:24.982209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:24.982209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:24.982209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:24.982209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:24.982209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:24.982209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:24.982209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:24.982209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:24.982209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:24.982209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:24.982209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:24.982209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:24.982209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:24.982209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:24.982209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:24.982209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:24.983007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:24.983007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:24.983007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:24.983007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:24.983007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:24.983007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:24.983007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:24.983007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:24.983007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:24.983007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:24.983007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:24.983007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:24.985757 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 170/341] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:05:24 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:24.993866 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642c4c14-9d26-4ad8-8164-6bdaf3901dd4 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:25.050092 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cffce90-f5a9-4e97-a6fb-62a3d6d85a94 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:25.103564 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71737ccc-ce74-45b0-a201-907a61b3ee49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:25.104584 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71737ccc-ce74-45b0-a201-907a61b3ee49 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:25.105168 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71737ccc-ce74-45b0-a201-907a61b3ee49 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:25.105653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71737ccc-ce74-45b0-a201-907a61b3ee49 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:25.121060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71737ccc-ce74-45b0-a201-907a61b3ee49 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:25.131911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-71737ccc-ce74-45b0-a201-907a61b3ee49 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:25.131911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:25.131911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:25.131911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:25.131911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:25.131911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:25.131911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:25.131911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:25.131911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:25.131911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:25.131911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:25.131911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:25.131911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:25.131911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:25.131911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:25.131911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:25.132796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:25.132796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:25.132796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:25.132796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:25.132796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:25.132796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:25.132796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:25.132796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:25.132796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:25.132796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:25.132796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:25.132796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:25.132796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:25.132796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:25.132796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:25.132796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:25.134125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:25.134125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:25.134125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:25.134125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:25.134125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:25.134125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:25.134125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:25.134125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:25.134125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:25.134125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:25.134125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:25.134125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:25.136754 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 172/342] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:25.147831 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd19aa4-484d-42b9-a2c7-bb20f80a1da7 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:25.148387 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd19aa4-484d-42b9-a2c7-bb20f80a1da7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:25.149425 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd19aa4-484d-42b9-a2c7-bb20f80a1da7 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:25.151246 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 173/343] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:25.152131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd19aa4-484d-42b9-a2c7-bb20f80a1da7 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:25.163737 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642c4c14-9d26-4ad8-8164-6bdaf3901dd4 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:25.171960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:25.205815 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5cfd9f-afcd-4e84-a7a3-b57cf8fac283 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:25.207309 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5cfd9f-afcd-4e84-a7a3-b57cf8fac283 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:25.207751 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5cfd9f-afcd-4e84-a7a3-b57cf8fac283 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:25.208181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5cfd9f-afcd-4e84-a7a3-b57cf8fac283 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:25.209012 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5cfd9f-afcd-4e84-a7a3-b57cf8fac283 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:25.209510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5cfd9f-afcd-4e84-a7a3-b57cf8fac283 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:25.211143 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5cfd9f-afcd-4e84-a7a3-b57cf8fac283 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:25.220506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 171/344] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:24 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:25.222015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2a0a20-1331-4fe5-b082-da29d5e0e5a4 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:25.222470 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2a0a20-1331-4fe5-b082-da29d5e0e5a4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:25.227054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2a0a20-1331-4fe5-b082-da29d5e0e5a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:25.255190 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-0e2a0a20-1331-4fe5-b082-da29d5e0e5a4 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:25.257040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0e2a0a20-1331-4fe5-b082-da29d5e0e5a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:25.310992 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:25.337012 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd19aa4-484d-42b9-a2c7-bb20f80a1da7 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:25.366793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 172/345] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:25.369290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2a0a20-1331-4fe5-b082-da29d5e0e5a4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:25.407390 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dea157-8ad2-4c54-bf4e-0255f0419fca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:25.407959 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dea157-8ad2-4c54-bf4e-0255f0419fca None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:25.408238 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dea157-8ad2-4c54-bf4e-0255f0419fca None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:25.408509 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dea157-8ad2-4c54-bf4e-0255f0419fca None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:25.409241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52dea157-8ad2-4c54-bf4e-0255f0419fca None admin] RBAC: Adding query filter params (name=member) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:25.409587 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52dea157-8ad2-4c54-bf4e-0255f0419fca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:25.411725 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52dea157-8ad2-4c54-bf4e-0255f0419fca None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:25.425808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 174/346] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:25.426962 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dea157-8ad2-4c54-bf4e-0255f0419fca None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:25.524872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:25.529845 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2a0a20-1331-4fe5-b082-da29d5e0e5a4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:25.556304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dea157-8ad2-4c54-bf4e-0255f0419fca None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:25.568554 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f399af1-d71e-43bb-a761-1d473c75b51a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:25.569402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f399af1-d71e-43bb-a761-1d473c75b51a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:25.569830 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f399af1-d71e-43bb-a761-1d473c75b51a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:25.570253 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f399af1-d71e-43bb-a761-1d473c75b51a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:25.591903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6079819d-c431-4b55-8f02-ad05ae49731c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:25.592775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6079819d-c431-4b55-8f02-ad05ae49731c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:25.593190 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6079819d-c431-4b55-8f02-ad05ae49731c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:25.593594 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6079819d-c431-4b55-8f02-ad05ae49731c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:25.604654 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6079819d-c431-4b55-8f02-ad05ae49731c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:25.611158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-6079819d-c431-4b55-8f02-ad05ae49731c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 01:05:25.611158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:25.611158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:25.611158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:25.611158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:25.611158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:25.611158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:25.611158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:25.611158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:25.611158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:25.611158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:25.611158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:25.611158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:25.611158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:25.611158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:25.611158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:25.612249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:25.612249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:25.612249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:25.612249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:25.612249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:25.612249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:25.612249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:25.612249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:25.612249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:25.612249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:25.612249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:25.612249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:25.612249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:25.612249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:25.612249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:25.612249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:25.613241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:25.613241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:25.613241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:25.613241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:25.613241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:25.613241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:25.613241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:25.613241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:25.613241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:25.613241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:25.613241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 01:05:25.613241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:25.615319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 175/347] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:25 2022] GET /identity/v3/roles/reader => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:25.616711 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e083d2-6ac9-494b-aed9-2b72e585c137 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:25.617132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e083d2-6ac9-494b-aed9-2b72e585c137 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:25.617625 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e083d2-6ac9-494b-aed9-2b72e585c137 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:25.619273 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 176/348] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:25.620097 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e083d2-6ac9-494b-aed9-2b72e585c137 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:25.632306 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:25.656834 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f399af1-d71e-43bb-a761-1d473c75b51a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:25.658820 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f399af1-d71e-43bb-a761-1d473c75b51a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:25.735782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e083d2-6ac9-494b-aed9-2b72e585c137 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:25.774593 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4832e9b9-0b35-4e03-8254-52b60adb1a28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:25.775648 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4832e9b9-0b35-4e03-8254-52b60adb1a28 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:25.775931 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4832e9b9-0b35-4e03-8254-52b60adb1a28 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:25.776204 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4832e9b9-0b35-4e03-8254-52b60adb1a28 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:25.777473 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:25.791071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 173/349] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:25.791903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f399af1-d71e-43bb-a761-1d473c75b51a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:25.866284 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4832e9b9-0b35-4e03-8254-52b60adb1a28 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:25.869434 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4832e9b9-0b35-4e03-8254-52b60adb1a28 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:25.878580 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:25.918809 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f399af1-d71e-43bb-a761-1d473c75b51a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:25.965467 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a81cf7-fbe4-42c6-bbb7-d48cb2f3af0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:25.966329 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a81cf7-fbe4-42c6-bbb7-d48cb2f3af0e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:25.966609 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a81cf7-fbe4-42c6-bbb7-d48cb2f3af0e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:25.970127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a81cf7-fbe4-42c6-bbb7-d48cb2f3af0e None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:25.970127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a81cf7-fbe4-42c6-bbb7-d48cb2f3af0e None admin] RBAC: Adding query filter params (name=reader) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:25.970127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a81cf7-fbe4-42c6-bbb7-d48cb2f3af0e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:25.970127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a81cf7-fbe4-42c6-bbb7-d48cb2f3af0e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:25.980163 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 174/350] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:25 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:25.981713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1855c7dd-6641-4770-8646-01c25969702f None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:25.981996 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1855c7dd-6641-4770-8646-01c25969702f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:25.982265 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1855c7dd-6641-4770-8646-01c25969702f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:26.005596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-1855c7dd-6641-4770-8646-01c25969702f None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:26.006779 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1855c7dd-6641-4770-8646-01c25969702f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:26.010386 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 177/351] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:26.015981 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4832e9b9-0b35-4e03-8254-52b60adb1a28 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:26.072892 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:26.090123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 175/352] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:26.095694 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1855c7dd-6641-4770-8646-01c25969702f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:26.136051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4832e9b9-0b35-4e03-8254-52b60adb1a28 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:26.151321 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:26.178737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006174cb-9ffa-4490-8167-34e63ca9a6e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:26.179546 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006174cb-9ffa-4490-8167-34e63ca9a6e8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:26.180437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006174cb-9ffa-4490-8167-34e63ca9a6e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:26.180815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006174cb-9ffa-4490-8167-34e63ca9a6e8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:26.195258 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006174cb-9ffa-4490-8167-34e63ca9a6e8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:26.200773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-006174cb-9ffa-4490-8167-34e63ca9a6e8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:26.200773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:26.200773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:26.200773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:26.200773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:26.200773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:26.200773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:26.200773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:26.200773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:26.200773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:26.200773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:26.200773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:26.200773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:26.200773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:26.200773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:26.200773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:26.201819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:26.201819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:26.201819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:26.201819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:26.201819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:26.201819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:26.201819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:26.201819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:26.201819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:26.201819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:26.201819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:26.201819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:26.201819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:26.201819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:26.201819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:26.201819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:26.202794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:26.202794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:26.202794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:26.202794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:26.202794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:26.202794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:26.202794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:26.202794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:26.202794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:26.202794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:26.202794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:26.202794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:26.209167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 178/353] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:26.210206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006174cb-9ffa-4490-8167-34e63ca9a6e8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:26.231274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1855c7dd-6641-4770-8646-01c25969702f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:26.267815 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5e0fad-a138-44ea-9ceb-9818ec622adc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:26.268474 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5e0fad-a138-44ea-9ceb-9818ec622adc None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:26.268728 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5e0fad-a138-44ea-9ceb-9818ec622adc None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:26.269042 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5e0fad-a138-44ea-9ceb-9818ec622adc None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:26.270172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5e0fad-a138-44ea-9ceb-9818ec622adc None admin] RBAC: Authorizing `identity:get_project(project_id=4426c2e7285b4cbc8331aa86690ef664)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:26.271705 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5e0fad-a138-44ea-9ceb-9818ec622adc None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:26.273480 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 176/354] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:26 2022] GET /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664 => generated 298 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:26.274808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5e0fad-a138-44ea-9ceb-9818ec622adc None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:26.343966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006174cb-9ffa-4490-8167-34e63ca9a6e8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:26.354947 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:26.392501 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e4cbfe-c607-43da-9477-92def0976257 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:26.393651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e4cbfe-c607-43da-9477-92def0976257 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:26.394438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e4cbfe-c607-43da-9477-92def0976257 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:26.395079 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e4cbfe-c607-43da-9477-92def0976257 None admin] PATH_INFO: `/v3/users/7b88d371cd594f4496ca02b449221e6f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:26.397657 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e4cbfe-c607-43da-9477-92def0976257 None admin] RBAC: Authorizing `identity:get_user(user_id=7b88d371cd594f4496ca02b449221e6f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:26.399918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e4cbfe-c607-43da-9477-92def0976257 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:26.402776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 179/355] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:05:26 2022] GET /identity/v3/users/7b88d371cd594f4496ca02b449221e6f => generated 293 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:26.403778 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5e0fad-a138-44ea-9ceb-9818ec622adc None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:26.404010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e4cbfe-c607-43da-9477-92def0976257 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:26.447979 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09328670-6375-464e-9c4e-d5c647561eb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:26.448541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09328670-6375-464e-9c4e-d5c647561eb0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:26.448826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09328670-6375-464e-9c4e-d5c647561eb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:26.449101 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09328670-6375-464e-9c4e-d5c647561eb0 None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:26.450214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09328670-6375-464e-9c4e-d5c647561eb0 None admin] RBAC: Authorizing `identity:get_project(project_id=4426c2e7285b4cbc8331aa86690ef664)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:26.451780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09328670-6375-464e-9c4e-d5c647561eb0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:26.451893 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:26.453425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 177/356] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:26 2022] GET /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664 => generated 298 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:26.454623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09328670-6375-464e-9c4e-d5c647561eb0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:26.548130 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e4cbfe-c607-43da-9477-92def0976257 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:26.580058 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09328670-6375-464e-9c4e-d5c647561eb0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:26.596033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f52c6ae-780c-43e5-89f6-f6ab22623a21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:26.596923 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f52c6ae-780c-43e5-89f6-f6ab22623a21 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:26.597346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f52c6ae-780c-43e5-89f6-f6ab22623a21 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:26.597776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f52c6ae-780c-43e5-89f6-f6ab22623a21 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:26.598836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f52c6ae-780c-43e5-89f6-f6ab22623a21 None admin] RBAC: Adding query filter params (name=member) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:26.599342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f52c6ae-780c-43e5-89f6-f6ab22623a21 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:26.600926 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f52c6ae-780c-43e5-89f6-f6ab22623a21 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:26.609876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 180/357] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:26.610023 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:26.610903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f52c6ae-780c-43e5-89f6-f6ab22623a21 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:26.619051 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c09ae6b-b52e-4ddc-84e5-4b0678379aa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:26.619788 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c09ae6b-b52e-4ddc-84e5-4b0678379aa6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:26.620538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c09ae6b-b52e-4ddc-84e5-4b0678379aa6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:26.620538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c09ae6b-b52e-4ddc-84e5-4b0678379aa6 None admin] PATH_INFO: `/v3/groups/790da853f67443aba550f1f2b09ca790` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:26.627473 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c09ae6b-b52e-4ddc-84e5-4b0678379aa6 None admin] RBAC: Authorizing `identity:get_group(group_id=790da853f67443aba550f1f2b09ca790)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:26.628926 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c09ae6b-b52e-4ddc-84e5-4b0678379aa6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:26.630896 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 178/358] 10.222.0.15 () {56 vars in 1180 bytes} [Tue Sep 13 01:05:26 2022] GET /identity/v3/groups/790da853f67443aba550f1f2b09ca790 => generated 226 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:26.631355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c09ae6b-b52e-4ddc-84e5-4b0678379aa6 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:26.724831 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:26.733935 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f52c6ae-780c-43e5-89f6-f6ab22623a21 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:26.748330 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c09ae6b-b52e-4ddc-84e5-4b0678379aa6 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:26.776934 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f70937-5e98-4ec4-8354-c157c59d074e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:26.777568 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f70937-5e98-4ec4-8354-c157c59d074e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:26.777841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f70937-5e98-4ec4-8354-c157c59d074e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:26.778106 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f70937-5e98-4ec4-8354-c157c59d074e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:26.788547 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748f5d3d-0b75-4de1-a881-d00cafb19ce0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:26.789150 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748f5d3d-0b75-4de1-a881-d00cafb19ce0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:26.789454 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748f5d3d-0b75-4de1-a881-d00cafb19ce0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:26.789847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748f5d3d-0b75-4de1-a881-d00cafb19ce0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:26.793231 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748f5d3d-0b75-4de1-a881-d00cafb19ce0 None admin] RBAC: Adding query filter params (role.id=6f98018cf28149aa84201e6dc4a43c2d, user.id=7b88d371cd594f4496ca02b449221e6f, scope.system=all) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:26.793231 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748f5d3d-0b75-4de1-a881-d00cafb19ce0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:26.793506 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f70937-5e98-4ec4-8354-c157c59d074e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:26.793637 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748f5d3d-0b75-4de1-a881-d00cafb19ce0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:26.797569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-96f70937-5e98-4ec4-8354-c157c59d074e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:26.797569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:26.797569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:26.797569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:26.797569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:26.797569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:26.797569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:26.797569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:26.797569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:26.797569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:26.797569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:26.797569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:26.797569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:26.797569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:26.797569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:26.797569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:26.798451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:26.798451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:26.798451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:26.798451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:26.798451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:26.798451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:26.798451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:26.798451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:26.798451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:26.798451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:26.798451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:26.798451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:26.798451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:26.798451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:26.798451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:26.798451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:26.799283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:26.799283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:26.799283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:26.799283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:26.799283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:26.799283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:26.799283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:26.799283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:26.799283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:26.799283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:26.799283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:26.799283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:26.799283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 181/359] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:05:26 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:26.800558 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c773ed-54fa-45fd-ab59-4a8251604add None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:26.800558 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c773ed-54fa-45fd-ab59-4a8251604add None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:26.800558 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c773ed-54fa-45fd-ab59-4a8251604add None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:26.803018 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 182/360] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:26.806220 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c773ed-54fa-45fd-ab59-4a8251604add None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:26.810971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 179/361] 10.222.0.15 () {56 vars in 1308 bytes} [Tue Sep 13 01:05:26 2022] GET /identity/v3/role_assignments?user.id=7b88d371cd594f4496ca02b449221e6f&scope.system=all&role.id=6f98018cf28149aa84201e6dc4a43c2d => generated 510 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:26.811885 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748f5d3d-0b75-4de1-a881-d00cafb19ce0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:26.894646 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:26.965020 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c773ed-54fa-45fd-ab59-4a8251604add None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:26.969182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748f5d3d-0b75-4de1-a881-d00cafb19ce0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:27.013594 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a043c73e-b85f-4658-97ce-85f5d1c1b5a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:27.014133 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a043c73e-b85f-4658-97ce-85f5d1c1b5a5 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:27.014409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a043c73e-b85f-4658-97ce-85f5d1c1b5a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:27.014774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a043c73e-b85f-4658-97ce-85f5d1c1b5a5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:27.015301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2c00a5-e5c3-4777-bb25-a9a38f54f636 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:27.016175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2c00a5-e5c3-4777-bb25-a9a38f54f636 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:27.016454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2c00a5-e5c3-4777-bb25-a9a38f54f636 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:27.016745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2c00a5-e5c3-4777-bb25-a9a38f54f636 None admin] PATH_INFO: `/v3/groups/790da853f67443aba550f1f2b09ca790` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:27.018020 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2c00a5-e5c3-4777-bb25-a9a38f54f636 None admin] RBAC: Authorizing `identity:get_group(group_id=790da853f67443aba550f1f2b09ca790)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:27.020423 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2c00a5-e5c3-4777-bb25-a9a38f54f636 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:27.022309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 183/362] 10.222.0.15 () {56 vars in 1180 bytes} [Tue Sep 13 01:05:26 2022] GET /identity/v3/groups/790da853f67443aba550f1f2b09ca790 => generated 226 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:27.023126 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2c00a5-e5c3-4777-bb25-a9a38f54f636 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:27.052145 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:27.107849 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a043c73e-b85f-4658-97ce-85f5d1c1b5a5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:27.112594 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a043c73e-b85f-4658-97ce-85f5d1c1b5a5 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:27.144254 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2c00a5-e5c3-4777-bb25-a9a38f54f636 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:27.167318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:27.187116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e3847e-0850-4bcb-ae41-7ba332c56511 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:27.187729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e3847e-0850-4bcb-ae41-7ba332c56511 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:27.188270 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e3847e-0850-4bcb-ae41-7ba332c56511 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:27.188545 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e3847e-0850-4bcb-ae41-7ba332c56511 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:27.189267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e3847e-0850-4bcb-ae41-7ba332c56511 None admin] RBAC: Adding query filter params (group.id=790da853f67443aba550f1f2b09ca790, role.id=cd616e816a394068867091116a3f1ce0, scope.project.id=4426c2e7285b4cbc8331aa86690ef664) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:27.189572 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e3847e-0850-4bcb-ae41-7ba332c56511 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:27.191032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e3847e-0850-4bcb-ae41-7ba332c56511 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:27.197300 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 184/363] 10.222.0.15 () {56 vars in 1376 bytes} [Tue Sep 13 01:05:27 2022] GET /identity/v3/role_assignments?group.id=790da853f67443aba550f1f2b09ca790&scope.project.id=4426c2e7285b4cbc8331aa86690ef664&role.id=cd616e816a394068867091116a3f1ce0 => generated 260 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:27.197932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e3847e-0850-4bcb-ae41-7ba332c56511 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:27.231688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 180/364] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:27.232946 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40026537-71f7-428f-8b02-35e0efc856de None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:27.233231 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40026537-71f7-428f-8b02-35e0efc856de None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:27.233498 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40026537-71f7-428f-8b02-35e0efc856de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:27.257976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-40026537-71f7-428f-8b02-35e0efc856de None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:27.259292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-40026537-71f7-428f-8b02-35e0efc856de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:27.321440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:27.336887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e3847e-0850-4bcb-ae41-7ba332c56511 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:27.352734 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 181/365] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:27.352924 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40026537-71f7-428f-8b02-35e0efc856de None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:27.390159 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93250c3-46db-48fe-99ab-e2575d2c4e90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:27.391643 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93250c3-46db-48fe-99ab-e2575d2c4e90 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:27.392121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93250c3-46db-48fe-99ab-e2575d2c4e90 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:27.392683 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93250c3-46db-48fe-99ab-e2575d2c4e90 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:27.393623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93250c3-46db-48fe-99ab-e2575d2c4e90 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:27.394124 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93250c3-46db-48fe-99ab-e2575d2c4e90 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:27.395978 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93250c3-46db-48fe-99ab-e2575d2c4e90 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:27.411455 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 185/366] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:27 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:27.412296 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93250c3-46db-48fe-99ab-e2575d2c4e90 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:27.476452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:27.506318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40026537-71f7-428f-8b02-35e0efc856de None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:27.529345 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93250c3-46db-48fe-99ab-e2575d2c4e90 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:27.550533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018d5817-7bf0-4f6c-b2c2-48d53f30ddb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:27.551174 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018d5817-7bf0-4f6c-b2c2-48d53f30ddb5 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:27.551449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018d5817-7bf0-4f6c-b2c2-48d53f30ddb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:27.552220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018d5817-7bf0-4f6c-b2c2-48d53f30ddb5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:27.552952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018d5817-7bf0-4f6c-b2c2-48d53f30ddb5 None admin] RBAC: Adding query filter params (group.id=790da853f67443aba550f1f2b09ca790, role.id=6f71034643cb4e8caee299f5d0bb863f, scope.project.id=4426c2e7285b4cbc8331aa86690ef664) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:27.553256 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018d5817-7bf0-4f6c-b2c2-48d53f30ddb5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:27.555508 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018d5817-7bf0-4f6c-b2c2-48d53f30ddb5 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:27.564061 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 182/367] 10.222.0.15 () {56 vars in 1376 bytes} [Tue Sep 13 01:05:27 2022] GET /identity/v3/role_assignments?group.id=790da853f67443aba550f1f2b09ca790&scope.project.id=4426c2e7285b4cbc8331aa86690ef664&role.id=6f71034643cb4e8caee299f5d0bb863f => generated 260 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:27.565150 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018d5817-7bf0-4f6c-b2c2-48d53f30ddb5 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:27.572155 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2ca55c-6d00-4ddb-bdc4-2649d2372989 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:27.573182 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2ca55c-6d00-4ddb-bdc4-2649d2372989 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:27.573686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2ca55c-6d00-4ddb-bdc4-2649d2372989 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:27.574161 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2ca55c-6d00-4ddb-bdc4-2649d2372989 None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:27.575935 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2ca55c-6d00-4ddb-bdc4-2649d2372989 None admin] RBAC: Authorizing `identity:get_project(project_id=16a6348ce8d34348b34d853f76445b3a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:27.577607 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2ca55c-6d00-4ddb-bdc4-2649d2372989 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:27.580205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 186/368] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:27 2022] GET /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a => generated 302 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:27.588298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2ca55c-6d00-4ddb-bdc4-2649d2372989 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:27.616338 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:27.689511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018d5817-7bf0-4f6c-b2c2-48d53f30ddb5 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:27.720712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2ca55c-6d00-4ddb-bdc4-2649d2372989 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:27.732898 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e61a96e-6b91-4388-99d4-374bcb8ccbce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:27.733913 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e61a96e-6b91-4388-99d4-374bcb8ccbce None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:27.734404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e61a96e-6b91-4388-99d4-374bcb8ccbce None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:27.734929 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e61a96e-6b91-4388-99d4-374bcb8ccbce None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:27.749710 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:27.763300 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23da9334-c647-4a88-b179-9163df6ce596 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:27.763957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23da9334-c647-4a88-b179-9163df6ce596 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:27.764237 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23da9334-c647-4a88-b179-9163df6ce596 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:27.764505 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23da9334-c647-4a88-b179-9163df6ce596 None admin] PATH_INFO: `/v3/groups/790da853f67443aba550f1f2b09ca790` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:27.765684 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23da9334-c647-4a88-b179-9163df6ce596 None admin] RBAC: Authorizing `identity:get_group(group_id=790da853f67443aba550f1f2b09ca790)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:27.768121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23da9334-c647-4a88-b179-9163df6ce596 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:27.770103 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 187/369] 10.222.0.15 () {56 vars in 1180 bytes} [Tue Sep 13 01:05:27 2022] GET /identity/v3/groups/790da853f67443aba550f1f2b09ca790 => generated 226 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:27.770958 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23da9334-c647-4a88-b179-9163df6ce596 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:27.827695 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e61a96e-6b91-4388-99d4-374bcb8ccbce None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:27.829407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e61a96e-6b91-4388-99d4-374bcb8ccbce None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:27.876984 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:27.897075 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23da9334-c647-4a88-b179-9163df6ce596 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:27.940167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8dc46c-9619-49a3-ab45-c8203999f054 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:27.940846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8dc46c-9619-49a3-ab45-c8203999f054 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:27.941114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8dc46c-9619-49a3-ab45-c8203999f054 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:27.941384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8dc46c-9619-49a3-ab45-c8203999f054 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:27.956366 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8dc46c-9619-49a3-ab45-c8203999f054 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:27.962037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-1b8dc46c-9619-49a3-ab45-c8203999f054 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:05:27.962037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:27.962037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:27.962037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:27.962037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:27.962037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:27.962037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:27.962037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:27.962037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:27.962037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:27.962037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:27.962037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:27.962037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:27.962037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:27.962037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:27.962037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:27.963085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:27.963085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:27.963085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:27.963085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:27.963085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:27.963085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:27.963085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:27.963085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:27.963085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:27.963085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:27.963085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:27.963085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:27.963085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:27.963085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:27.963085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:27.963085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:27.964029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:27.964029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:27.964029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:27.964029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:27.964029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:27.964029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:27.964029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:27.964029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:27.964029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:27.964029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:27.964029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:05:27.964029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:27.964029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 188/370] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:05:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:27.964721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8dc46c-9619-49a3-ab45-c8203999f054 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:27.970722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 183/371] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:27.975608 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e61a96e-6b91-4388-99d4-374bcb8ccbce None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:28.030974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:28.105494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8dc46c-9619-49a3-ab45-c8203999f054 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:28.113546 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e61a96e-6b91-4388-99d4-374bcb8ccbce None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:28.147019 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa135ff-e762-4fea-bd46-f67adf9157a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:28.148005 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa135ff-e762-4fea-bd46-f67adf9157a8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:28.148504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa135ff-e762-4fea-bd46-f67adf9157a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:28.149051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa135ff-e762-4fea-bd46-f67adf9157a8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:28.149973 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa135ff-e762-4fea-bd46-f67adf9157a8 None admin] RBAC: Adding query filter params (group.id=790da853f67443aba550f1f2b09ca790, role.id=6f71034643cb4e8caee299f5d0bb863f, scope.project.id=16a6348ce8d34348b34d853f76445b3a) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:28.150491 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa135ff-e762-4fea-bd46-f67adf9157a8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:28.152187 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa135ff-e762-4fea-bd46-f67adf9157a8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:28.158599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 189/372] 10.222.0.15 () {56 vars in 1376 bytes} [Tue Sep 13 01:05:27 2022] GET /identity/v3/role_assignments?group.id=790da853f67443aba550f1f2b09ca790&scope.project.id=16a6348ce8d34348b34d853f76445b3a&role.id=6f71034643cb4e8caee299f5d0bb863f => generated 260 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:28.159659 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa135ff-e762-4fea-bd46-f67adf9157a8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:28.161903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d68801d-34b0-4c96-884c-c309a9f4f6ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:28.162579 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d68801d-34b0-4c96-884c-c309a9f4f6ce None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:28.162892 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d68801d-34b0-4c96-884c-c309a9f4f6ce None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:28.163151 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d68801d-34b0-4c96-884c-c309a9f4f6ce None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:28.163830 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d68801d-34b0-4c96-884c-c309a9f4f6ce None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:28.164197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d68801d-34b0-4c96-884c-c309a9f4f6ce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:28.166515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d68801d-34b0-4c96-884c-c309a9f4f6ce None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:28.175695 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 184/373] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:28.176291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:28.181623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d68801d-34b0-4c96-884c-c309a9f4f6ce None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:28.278847 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa135ff-e762-4fea-bd46-f67adf9157a8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:28.318403 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20f8b28-57cb-4694-bd73-b676a53eae2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:28.319436 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20f8b28-57cb-4694-bd73-b676a53eae2f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:28.319944 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20f8b28-57cb-4694-bd73-b676a53eae2f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:28.320426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20f8b28-57cb-4694-bd73-b676a53eae2f None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:28.322433 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20f8b28-57cb-4694-bd73-b676a53eae2f None admin] RBAC: Authorizing `identity:get_project(project_id=16a6348ce8d34348b34d853f76445b3a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:28.324131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20f8b28-57cb-4694-bd73-b676a53eae2f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:28.326375 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 190/374] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:28 2022] GET /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a => generated 302 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:28.336029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20f8b28-57cb-4694-bd73-b676a53eae2f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:28.373306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d68801d-34b0-4c96-884c-c309a9f4f6ce None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:28.379766 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:28.446294 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45f8398-d985-47b0-9dc9-c7107af40538 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:28.446928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45f8398-d985-47b0-9dc9-c7107af40538 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:28.447207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45f8398-d985-47b0-9dc9-c7107af40538 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:28.447474 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45f8398-d985-47b0-9dc9-c7107af40538 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:28.493715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:28.533347 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20f8b28-57cb-4694-bd73-b676a53eae2f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:28.568952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45f8398-d985-47b0-9dc9-c7107af40538 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:28.570401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45f8398-d985-47b0-9dc9-c7107af40538 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:28.596703 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17764783-2712-4d4d-8454-f0b4b5bdb325 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:28.598070 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17764783-2712-4d4d-8454-f0b4b5bdb325 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:28.598353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17764783-2712-4d4d-8454-f0b4b5bdb325 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:28.598687 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17764783-2712-4d4d-8454-f0b4b5bdb325 None admin] PATH_INFO: `/v3/groups/790da853f67443aba550f1f2b09ca790` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:28.602777 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17764783-2712-4d4d-8454-f0b4b5bdb325 None admin] RBAC: Authorizing `identity:get_group(group_id=790da853f67443aba550f1f2b09ca790)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:28.604283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17764783-2712-4d4d-8454-f0b4b5bdb325 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:28.610923 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 191/375] 10.222.0.15 () {56 vars in 1180 bytes} [Tue Sep 13 01:05:28 2022] GET /identity/v3/groups/790da853f67443aba550f1f2b09ca790 => generated 226 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:28.618243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17764783-2712-4d4d-8454-f0b4b5bdb325 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:28.753690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 185/376] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 573 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 13 01:05:28.759525 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45f8398-d985-47b0-9dc9-c7107af40538 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:28.764219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:28.769823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17764783-2712-4d4d-8454-f0b4b5bdb325 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:28.817617 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7fd018-31a5-40f5-b54b-aa655892a5f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:28.818251 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7fd018-31a5-40f5-b54b-aa655892a5f6 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:28.818524 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7fd018-31a5-40f5-b54b-aa655892a5f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:28.818792 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7fd018-31a5-40f5-b54b-aa655892a5f6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:28.819547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7fd018-31a5-40f5-b54b-aa655892a5f6 None admin] RBAC: Adding query filter params (group.id=790da853f67443aba550f1f2b09ca790, role.id=cd616e816a394068867091116a3f1ce0, scope.project.id=16a6348ce8d34348b34d853f76445b3a) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:28.819852 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7fd018-31a5-40f5-b54b-aa655892a5f6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:28.824165 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7fd018-31a5-40f5-b54b-aa655892a5f6 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:28.843308 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 192/377] 10.222.0.15 () {56 vars in 1376 bytes} [Tue Sep 13 01:05:28 2022] GET /identity/v3/role_assignments?group.id=790da853f67443aba550f1f2b09ca790&scope.project.id=16a6348ce8d34348b34d853f76445b3a&role.id=cd616e816a394068867091116a3f1ce0 => generated 260 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:28.926308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45f8398-d985-47b0-9dc9-c7107af40538 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:28.973061 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aec46df-6662-4376-9e6e-dc2522590639 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:28.973725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aec46df-6662-4376-9e6e-dc2522590639 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:28.974001 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aec46df-6662-4376-9e6e-dc2522590639 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:28.974271 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aec46df-6662-4376-9e6e-dc2522590639 None admin] PATH_INFO: `/v3/projects/ce37ba2e4948471599c41255ecf54622` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:28.975540 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aec46df-6662-4376-9e6e-dc2522590639 None admin] RBAC: Authorizing `identity:get_project(project_id=ce37ba2e4948471599c41255ecf54622)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:28.977570 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aec46df-6662-4376-9e6e-dc2522590639 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:28.979286 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 186/378] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:28 2022] GET /identity/v3/projects/ce37ba2e4948471599c41255ecf54622 => generated 344 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:28.985129 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7fd018-31a5-40f5-b54b-aa655892a5f6 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:29.002914 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:29.114548 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7fd018-31a5-40f5-b54b-aa655892a5f6 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:29.156465 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad73841f-f435-4f34-a58e-4263070b1452 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:29.157111 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad73841f-f435-4f34-a58e-4263070b1452 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:29.157387 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad73841f-f435-4f34-a58e-4263070b1452 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:29.157683 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad73841f-f435-4f34-a58e-4263070b1452 None admin] PATH_INFO: `/v3/groups/4b30643760554c6fa0e17ed2e13e5aad` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:29.169312 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad73841f-f435-4f34-a58e-4263070b1452 None admin] RBAC: Authorizing `identity:get_group(group_id=4b30643760554c6fa0e17ed2e13e5aad)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:29.170771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad73841f-f435-4f34-a58e-4263070b1452 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:29.172751 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 193/379] 10.222.0.15 () {56 vars in 1180 bytes} [Tue Sep 13 01:05:28 2022] GET /identity/v3/groups/4b30643760554c6fa0e17ed2e13e5aad => generated 229 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:29.178416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aec46df-6662-4376-9e6e-dc2522590639 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:29.303230 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:29.345767 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aec46df-6662-4376-9e6e-dc2522590639 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:29.396212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fa9543-6850-4bca-a02e-9d8ee2bee4be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:29.396843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fa9543-6850-4bca-a02e-9d8ee2bee4be None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:29.397063 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fa9543-6850-4bca-a02e-9d8ee2bee4be None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:29.397269 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fa9543-6850-4bca-a02e-9d8ee2bee4be None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:29.397998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fa9543-6850-4bca-a02e-9d8ee2bee4be None admin] RBAC: Adding query filter params (group.id=4b30643760554c6fa0e17ed2e13e5aad, role.id=66a36c858aab40688080eed6ceace3a0, scope.project.id=ce37ba2e4948471599c41255ecf54622) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:29.398306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fa9543-6850-4bca-a02e-9d8ee2bee4be None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:29.399869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fa9543-6850-4bca-a02e-9d8ee2bee4be None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:29.406013 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 187/380] 10.222.0.15 () {56 vars in 1376 bytes} [Tue Sep 13 01:05:29 2022] GET /identity/v3/role_assignments?group.id=4b30643760554c6fa0e17ed2e13e5aad&scope.project.id=ce37ba2e4948471599c41255ecf54622&role.id=66a36c858aab40688080eed6ceace3a0 => generated 260 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:29.755609 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15a63e3-cb97-4bdd-990d-b515821516af None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:29.756121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15a63e3-cb97-4bdd-990d-b515821516af None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:29.756195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15a63e3-cb97-4bdd-990d-b515821516af None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:29.757701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 194/381] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:29.763099 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7ccc93-26f3-4d23-bdd6-3e5429301567 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:29.763386 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7ccc93-26f3-4d23-bdd6-3e5429301567 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:29.763662 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7ccc93-26f3-4d23-bdd6-3e5429301567 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:29.791887 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-5b7ccc93-26f3-4d23-bdd6-3e5429301567 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:29.793092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5b7ccc93-26f3-4d23-bdd6-3e5429301567 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:29.877104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 188/382] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:30.277900 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15a63e3-cb97-4bdd-990d-b515821516af None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:30.379984 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:30.416315 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15a63e3-cb97-4bdd-990d-b515821516af None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:30.471343 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efcfa32c-a484-46ef-90aa-b68b319bb183 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:30.471975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcfa32c-a484-46ef-90aa-b68b319bb183 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:30.472247 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcfa32c-a484-46ef-90aa-b68b319bb183 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:30.472513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efcfa32c-a484-46ef-90aa-b68b319bb183 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:30.484212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7fb9863-2e1e-405b-9ae3-5d2ef8b9871d None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:30.484706 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7fb9863-2e1e-405b-9ae3-5d2ef8b9871d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:30.484798 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7fb9863-2e1e-405b-9ae3-5d2ef8b9871d None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:30.486247 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 189/383] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:30.491043 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efcfa32c-a484-46ef-90aa-b68b319bb183 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:30.497086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-efcfa32c-a484-46ef-90aa-b68b319bb183 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:30.497086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:30.497086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:30.497086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:30.497086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:30.497086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:30.497086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:30.497086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:30.497086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:30.497086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:30.497086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:30.497086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:30.497086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:30.497086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:30.497086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:30.497086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:30.497948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:30.497948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:30.497948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:30.497948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:30.497948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:30.497948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:30.497948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:30.497948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:30.497948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:30.497948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:30.497948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:30.497948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:30.497948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:30.497948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:30.497948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:30.497948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:30.498722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:30.498722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:30.498722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:30.498722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:30.498722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:30.498722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:30.498722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:30.498722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:30.498722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:30.498722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:30.498722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:30.498722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:30.500985 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 195/384] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:05:30 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:30.507874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7fb9863-2e1e-405b-9ae3-5d2ef8b9871d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:30.510825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6053b7-baac-46c8-8003-a0179e28a2bb None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:30.513065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6053b7-baac-46c8-8003-a0179e28a2bb None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:30.513566 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6053b7-baac-46c8-8003-a0179e28a2bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:30.546816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-2f6053b7-baac-46c8-8003-a0179e28a2bb None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:30.551320 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2f6053b7-baac-46c8-8003-a0179e28a2bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:30.628547 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:30.673869 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 196/385] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:30.718461 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7fb9863-2e1e-405b-9ae3-5d2ef8b9871d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:30.766240 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bff4872-fe20-44af-bd79-993d758bac92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:30.766842 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bff4872-fe20-44af-bd79-993d758bac92 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:30.767148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bff4872-fe20-44af-bd79-993d758bac92 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:30.767418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bff4872-fe20-44af-bd79-993d758bac92 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:30.768078 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bff4872-fe20-44af-bd79-993d758bac92 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:30.768377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bff4872-fe20-44af-bd79-993d758bac92 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:30.769747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bff4872-fe20-44af-bd79-993d758bac92 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:30.778174 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 190/386] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:30 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:30.784240 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6053b7-baac-46c8-8003-a0179e28a2bb None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:30.907171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:30.908315 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b9a152-f669-4121-b028-005126b16cb7 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:30.908624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b9a152-f669-4121-b028-005126b16cb7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:30.908907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b9a152-f669-4121-b028-005126b16cb7 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:30.910225 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 191/387] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:30.916540 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd907d44-e92e-4a26-abd9-99fbd1b8d1ff None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:30.916816 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd907d44-e92e-4a26-abd9-99fbd1b8d1ff None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:30.917088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd907d44-e92e-4a26-abd9-99fbd1b8d1ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:30.947415 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-bd907d44-e92e-4a26-abd9-99fbd1b8d1ff None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:30.948607 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-bd907d44-e92e-4a26-abd9-99fbd1b8d1ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:30.988390 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6053b7-baac-46c8-8003-a0179e28a2bb None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:31.026472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce727dcb-8b66-452a-8499-3f0d8aabe1f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:31.027138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce727dcb-8b66-452a-8499-3f0d8aabe1f4 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:31.027420 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce727dcb-8b66-452a-8499-3f0d8aabe1f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:31.027693 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce727dcb-8b66-452a-8499-3f0d8aabe1f4 None admin] PATH_INFO: `/v3/groups/790da853f67443aba550f1f2b09ca790` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:31.028790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce727dcb-8b66-452a-8499-3f0d8aabe1f4 None admin] RBAC: Authorizing `identity:get_group(group_id=790da853f67443aba550f1f2b09ca790)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:31.030214 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce727dcb-8b66-452a-8499-3f0d8aabe1f4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:31.032101 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 197/388] 10.222.0.15 () {56 vars in 1180 bytes} [Tue Sep 13 01:05:30 2022] GET /identity/v3/groups/790da853f67443aba550f1f2b09ca790 => generated 226 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:31.032858 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce727dcb-8b66-452a-8499-3f0d8aabe1f4 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:31.044065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 192/389] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 01:05:31.045339 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd907d44-e92e-4a26-abd9-99fbd1b8d1ff None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:31.148319 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:31.156722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce727dcb-8b66-452a-8499-3f0d8aabe1f4 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:31.197890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:31.203952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd907d44-e92e-4a26-abd9-99fbd1b8d1ff None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:31.206879 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a560db5a-68cb-42d0-98f7-63555bd63050 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:31.207530 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a560db5a-68cb-42d0-98f7-63555bd63050 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:31.207807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a560db5a-68cb-42d0-98f7-63555bd63050 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:31.208079 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a560db5a-68cb-42d0-98f7-63555bd63050 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:31.231409 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a560db5a-68cb-42d0-98f7-63555bd63050 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:31.236537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-a560db5a-68cb-42d0-98f7-63555bd63050 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:31.236537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:31.236537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:31.236537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:31.236537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:31.236537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:31.236537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:31.236537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:31.236537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:31.236537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:31.236537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:31.236537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:31.236537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:31.236537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:31.236537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:31.236537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:31.237426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:31.237426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:31.237426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:31.237426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:31.237426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:31.237426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:31.237426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:31.237426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:31.237426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:31.237426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:31.237426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:31.237426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:31.237426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:31.237426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:31.237426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:31.237426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:31.238206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:31.238206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:31.238206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:31.238206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:31.238206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:31.238206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:31.238206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:31.238206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:31.238206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:31.238206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:31.238206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:31.238206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:31.239453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 198/390] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:31.243583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833a434e-48d8-440d-a902-4040ba14300b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:31.244406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833a434e-48d8-440d-a902-4040ba14300b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:31.244824 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833a434e-48d8-440d-a902-4040ba14300b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:31.245226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833a434e-48d8-440d-a902-4040ba14300b None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:31.246485 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833a434e-48d8-440d-a902-4040ba14300b None admin] RBAC: Authorizing `identity:get_project(project_id=4426c2e7285b4cbc8331aa86690ef664)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:31.247241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a560db5a-68cb-42d0-98f7-63555bd63050 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:31.248145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833a434e-48d8-440d-a902-4040ba14300b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:31.250145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 193/391] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:31 2022] GET /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664 => generated 298 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:31.260692 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833a434e-48d8-440d-a902-4040ba14300b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:31.374504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a560db5a-68cb-42d0-98f7-63555bd63050 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:31.421589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa995aa-6cc5-4b34-9e59-ff39a025f9fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:31.422203 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa995aa-6cc5-4b34-9e59-ff39a025f9fa None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:31.422474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa995aa-6cc5-4b34-9e59-ff39a025f9fa None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:31.422737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa995aa-6cc5-4b34-9e59-ff39a025f9fa None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:31.423418 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa995aa-6cc5-4b34-9e59-ff39a025f9fa None admin] RBAC: Adding query filter params (name=member) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:31.423715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa995aa-6cc5-4b34-9e59-ff39a025f9fa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:31.425073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa995aa-6cc5-4b34-9e59-ff39a025f9fa None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:31.442899 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 199/392] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:31.443515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833a434e-48d8-440d-a902-4040ba14300b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:31.444330 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa995aa-6cc5-4b34-9e59-ff39a025f9fa None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:31.475876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:31.485371 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3a1ee0-7f8d-40ba-8d8b-065ae0c37a93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:31.486130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3a1ee0-7f8d-40ba-8d8b-065ae0c37a93 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:31.486313 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3a1ee0-7f8d-40ba-8d8b-065ae0c37a93 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:31.486562 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3a1ee0-7f8d-40ba-8d8b-065ae0c37a93 None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664/groups/790da853f67443aba550f1f2b09ca790/roles/cd616e816a394068867091116a3f1ce0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:31.489078 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3a1ee0-7f8d-40ba-8d8b-065ae0c37a93 None admin] RBAC: Authorizing `identity:create_grant(project_id=4426c2e7285b4cbc8331aa86690ef664, group_id=790da853f67443aba550f1f2b09ca790, role_id=cd616e816a394068867091116a3f1ce0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:31.490655 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3a1ee0-7f8d-40ba-8d8b-065ae0c37a93 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:31.508344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 194/393] 10.222.0.15 () {58 vars in 1442 bytes} [Tue Sep 13 01:05:31 2022] PUT /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664/groups/790da853f67443aba550f1f2b09ca790/roles/cd616e816a394068867091116a3f1ce0 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:05:31.509267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3a1ee0-7f8d-40ba-8d8b-065ae0c37a93 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:31.524730 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:31.620286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa995aa-6cc5-4b34-9e59-ff39a025f9fa None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:31.661904 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9669e970-c63a-4783-baab-8ff3dac01b51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:31.662544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9669e970-c63a-4783-baab-8ff3dac01b51 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:31.662822 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9669e970-c63a-4783-baab-8ff3dac01b51 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:31.663126 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9669e970-c63a-4783-baab-8ff3dac01b51 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:31.675942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9669e970-c63a-4783-baab-8ff3dac01b51 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:31.677223 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3a1ee0-7f8d-40ba-8d8b-065ae0c37a93 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:31.681673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-9669e970-c63a-4783-baab-8ff3dac01b51 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:31.681673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:31.681673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:31.681673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:31.681673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:31.681673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:31.681673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:31.681673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:31.681673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:31.681673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:31.681673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:31.681673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:31.681673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:31.681673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:31.681673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:31.681673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:31.682788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:31.682788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:31.682788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:31.682788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:31.682788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:31.682788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:31.682788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:31.682788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:31.682788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:31.682788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:31.682788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:31.682788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:31.682788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:31.682788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:31.682788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:31.682788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:31.683803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:31.683803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:31.683803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:31.683803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:31.683803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:31.683803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:31.683803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:31.683803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:31.683803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:31.683803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:31.683803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:31.683803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:31.688178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 200/394] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:31.689743 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867f04f3-d12d-4c15-b8ff-6a7407c1a0c9 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:31.689863 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867f04f3-d12d-4c15-b8ff-6a7407c1a0c9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:31.690118 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867f04f3-d12d-4c15-b8ff-6a7407c1a0c9 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:31.696251 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 201/395] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:31 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:31.697101 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867f04f3-d12d-4c15-b8ff-6a7407c1a0c9 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:31.724004 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e66fae-7fac-4d6c-8449-215ac1725049 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:31.724769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e66fae-7fac-4d6c-8449-215ac1725049 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:31.725050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e66fae-7fac-4d6c-8449-215ac1725049 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:31.725316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e66fae-7fac-4d6c-8449-215ac1725049 None admin] PATH_INFO: `/v3/groups/790da853f67443aba550f1f2b09ca790` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:31.727732 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e66fae-7fac-4d6c-8449-215ac1725049 None admin] RBAC: Authorizing `identity:get_group(group_id=790da853f67443aba550f1f2b09ca790)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:31.729273 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e66fae-7fac-4d6c-8449-215ac1725049 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:31.734966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 195/396] 10.222.0.15 () {56 vars in 1180 bytes} [Tue Sep 13 01:05:31 2022] GET /identity/v3/groups/790da853f67443aba550f1f2b09ca790 => generated 226 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:31.734966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e756c74-0852-48a1-94be-9f606d6fa45f None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:31.734966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e756c74-0852-48a1-94be-9f606d6fa45f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:31.734966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e756c74-0852-48a1-94be-9f606d6fa45f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:31.750798 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-0e756c74-0852-48a1-94be-9f606d6fa45f None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:31.752143 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0e756c74-0852-48a1-94be-9f606d6fa45f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:31.814953 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:31.827069 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867f04f3-d12d-4c15-b8ff-6a7407c1a0c9 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:31.842165 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 196/397] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:31.846951 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e756c74-0852-48a1-94be-9f606d6fa45f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:31.853003 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:31.908882 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687cf765-2d96-4cf2-9cbe-e366d595cb6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:31.910724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687cf765-2d96-4cf2-9cbe-e366d595cb6f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:31.912406 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687cf765-2d96-4cf2-9cbe-e366d595cb6f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:31.912897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687cf765-2d96-4cf2-9cbe-e366d595cb6f None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:31.915439 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687cf765-2d96-4cf2-9cbe-e366d595cb6f None admin] RBAC: Adding query filter params (name=member) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:31.915973 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687cf765-2d96-4cf2-9cbe-e366d595cb6f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:31.917779 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687cf765-2d96-4cf2-9cbe-e366d595cb6f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:31.936482 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 202/398] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:31.944714 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687cf765-2d96-4cf2-9cbe-e366d595cb6f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:32.001728 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e756c74-0852-48a1-94be-9f606d6fa45f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:32.066009 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721ce71c-ba1c-4f5f-b437-981a3e1a4d6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:32.068123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721ce71c-ba1c-4f5f-b437-981a3e1a4d6b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:32.069572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721ce71c-ba1c-4f5f-b437-981a3e1a4d6b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:32.070057 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721ce71c-ba1c-4f5f-b437-981a3e1a4d6b None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:32.075694 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721ce71c-ba1c-4f5f-b437-981a3e1a4d6b None admin] RBAC: Authorizing `identity:get_project(project_id=4426c2e7285b4cbc8331aa86690ef664)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:32.077409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721ce71c-ba1c-4f5f-b437-981a3e1a4d6b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:32.080561 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 197/399] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:31 2022] GET /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664 => generated 298 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:32.081880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977a01bb-ac35-42a2-bbca-5d2760d1dd85 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:32.082168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977a01bb-ac35-42a2-bbca-5d2760d1dd85 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:32.082444 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977a01bb-ac35-42a2-bbca-5d2760d1dd85 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:32.084084 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 198/400] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:32.088793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977a01bb-ac35-42a2-bbca-5d2760d1dd85 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:32.094198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687cf765-2d96-4cf2-9cbe-e366d595cb6f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:32.132650 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e71671-7a39-4faf-8813-24e8ed372a46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:32.133915 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e71671-7a39-4faf-8813-24e8ed372a46 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:32.134191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e71671-7a39-4faf-8813-24e8ed372a46 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:32.134458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e71671-7a39-4faf-8813-24e8ed372a46 None admin] PATH_INFO: `/v3/groups/790da853f67443aba550f1f2b09ca790` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:32.143354 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e71671-7a39-4faf-8813-24e8ed372a46 None admin] RBAC: Authorizing `identity:get_group(group_id=790da853f67443aba550f1f2b09ca790)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:32.143354 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e71671-7a39-4faf-8813-24e8ed372a46 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:32.143354 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 203/401] 10.222.0.15 () {56 vars in 1180 bytes} [Tue Sep 13 01:05:31 2022] GET /identity/v3/groups/790da853f67443aba550f1f2b09ca790 => generated 226 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:32.143633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855c409a-5150-412c-a5d7-6298f533cb78 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:32.143817 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855c409a-5150-412c-a5d7-6298f533cb78 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:32.144087 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855c409a-5150-412c-a5d7-6298f533cb78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:32.161637 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-855c409a-5150-412c-a5d7-6298f533cb78 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:32.162794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-855c409a-5150-412c-a5d7-6298f533cb78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:32.166956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:32.205492 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:32.217729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977a01bb-ac35-42a2-bbca-5d2760d1dd85 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:32.234562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 204/402] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:32.235669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855c409a-5150-412c-a5d7-6298f533cb78 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:32.266346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4086e5e9-2df8-46a0-85d7-ecf436ffe522 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:32.267643 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4086e5e9-2df8-46a0-85d7-ecf436ffe522 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:32.268515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4086e5e9-2df8-46a0-85d7-ecf436ffe522 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:32.269249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4086e5e9-2df8-46a0-85d7-ecf436ffe522 None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664/groups/790da853f67443aba550f1f2b09ca790/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:32.271676 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4086e5e9-2df8-46a0-85d7-ecf436ffe522 None admin] RBAC: Authorizing `identity:create_grant(project_id=4426c2e7285b4cbc8331aa86690ef664, group_id=790da853f67443aba550f1f2b09ca790, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:32.274710 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4086e5e9-2df8-46a0-85d7-ecf436ffe522 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:32.297434 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 199/403] 10.222.0.15 () {58 vars in 1442 bytes} [Tue Sep 13 01:05:32 2022] PUT /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664/groups/790da853f67443aba550f1f2b09ca790/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:05:32.299353 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4086e5e9-2df8-46a0-85d7-ecf436ffe522 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:32.377550 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855c409a-5150-412c-a5d7-6298f533cb78 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:32.472017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4086e5e9-2df8-46a0-85d7-ecf436ffe522 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:32.480598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799ea5f2-ad50-41eb-aea3-0e54c97b95ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:32.481247 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799ea5f2-ad50-41eb-aea3-0e54c97b95ec None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:32.481521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799ea5f2-ad50-41eb-aea3-0e54c97b95ec None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:32.481789 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799ea5f2-ad50-41eb-aea3-0e54c97b95ec None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:32.483595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799ea5f2-ad50-41eb-aea3-0e54c97b95ec None admin] RBAC: Authorizing `identity:get_project(project_id=16a6348ce8d34348b34d853f76445b3a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:32.485488 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799ea5f2-ad50-41eb-aea3-0e54c97b95ec None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:32.492302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 205/404] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:32 2022] GET /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a => generated 302 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:32.496185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:32.498659 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799ea5f2-ad50-41eb-aea3-0e54c97b95ec None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:32.553681 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c37b76-e044-42d6-9bf7-48d97eaa81b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:32.554232 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c37b76-e044-42d6-9bf7-48d97eaa81b8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:32.554507 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c37b76-e044-42d6-9bf7-48d97eaa81b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:32.554810 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c37b76-e044-42d6-9bf7-48d97eaa81b8 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:32.559097 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:32.568512 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c37b76-e044-42d6-9bf7-48d97eaa81b8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:32.572843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-e8c37b76-e044-42d6-9bf7-48d97eaa81b8 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:32.572843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:32.572843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:32.572843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:32.572843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:32.572843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:32.572843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:32.572843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:32.572843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:32.572843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:32.572843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:32.572843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:32.572843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:32.572843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:32.572843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:32.572843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:32.573720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:32.573720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:32.573720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:32.573720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:32.573720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:32.573720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:32.573720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:32.573720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:32.573720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:32.573720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:32.573720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:32.573720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:32.573720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:32.573720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:32.573720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:32.573720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:32.576071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:32.576071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:32.576071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:32.576071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:32.576071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:32.576071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:32.576071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:32.576071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:32.576071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:32.576071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:32.576071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:32.576071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:32.576071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 200/405] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:05:32 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:32.576071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c37b76-e044-42d6-9bf7-48d97eaa81b8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:32.638995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799ea5f2-ad50-41eb-aea3-0e54c97b95ec None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:32.680800 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c37b76-e044-42d6-9bf7-48d97eaa81b8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:32.682493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aac9db6-0580-4109-a350-31373170b687 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:32.684303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac9db6-0580-4109-a350-31373170b687 None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:32.684790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac9db6-0580-4109-a350-31373170b687 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:32.685310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac9db6-0580-4109-a350-31373170b687 None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a/groups/790da853f67443aba550f1f2b09ca790/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:32.687448 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aac9db6-0580-4109-a350-31373170b687 None admin] RBAC: Authorizing `identity:create_grant(project_id=16a6348ce8d34348b34d853f76445b3a, group_id=790da853f67443aba550f1f2b09ca790, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:32.689889 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aac9db6-0580-4109-a350-31373170b687 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:32.714942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 206/406] 10.222.0.15 () {58 vars in 1442 bytes} [Tue Sep 13 01:05:32 2022] PUT /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a/groups/790da853f67443aba550f1f2b09ca790/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:05:32.716264 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aac9db6-0580-4109-a350-31373170b687 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:32.723300 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f34d74c-7865-4a09-880b-d0036499bc05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:32.724002 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f34d74c-7865-4a09-880b-d0036499bc05 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:32.724324 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f34d74c-7865-4a09-880b-d0036499bc05 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:32.724596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f34d74c-7865-4a09-880b-d0036499bc05 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:32.747218 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f34d74c-7865-4a09-880b-d0036499bc05 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:32.752463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-5f34d74c-7865-4a09-880b-d0036499bc05 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:05:32.752463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:32.752463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:32.752463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:32.752463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:32.752463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:32.752463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:32.752463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:32.752463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:32.752463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:32.752463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:32.752463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:32.752463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:32.752463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:32.752463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:32.752463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:32.753506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:32.753506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:32.753506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:32.753506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:32.753506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:32.753506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:32.753506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:32.753506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:32.753506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:32.753506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:32.753506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:32.753506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:32.753506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:32.753506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:32.753506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:32.753506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:32.754802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:32.754802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:32.754802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:32.754802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:32.754802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:32.754802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:32.754802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:32.754802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:32.754802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:32.754802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:32.754802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:05:32.754802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:32.754802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 201/407] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:05:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:32.763124 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f34d74c-7865-4a09-880b-d0036499bc05 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:32.823836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:32.949240 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aac9db6-0580-4109-a350-31373170b687 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:32.953193 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:32.964683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f34d74c-7865-4a09-880b-d0036499bc05 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:33.005987 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51cd4f5-c73d-4fbb-87d1-fa659321c96a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:33.006597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51cd4f5-c73d-4fbb-87d1-fa659321c96a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:33.006918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51cd4f5-c73d-4fbb-87d1-fa659321c96a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:33.007160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51cd4f5-c73d-4fbb-87d1-fa659321c96a None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:33.007816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51cd4f5-c73d-4fbb-87d1-fa659321c96a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:33.008185 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51cd4f5-c73d-4fbb-87d1-fa659321c96a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:33.009532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51cd4f5-c73d-4fbb-87d1-fa659321c96a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:33.012239 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39532a05-50a1-4a28-8a9f-b2a212c757aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:33.013089 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39532a05-50a1-4a28-8a9f-b2a212c757aa None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:33.013576 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39532a05-50a1-4a28-8a9f-b2a212c757aa None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:33.014044 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39532a05-50a1-4a28-8a9f-b2a212c757aa None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:33.015039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39532a05-50a1-4a28-8a9f-b2a212c757aa None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:33.015539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39532a05-50a1-4a28-8a9f-b2a212c757aa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:33.017112 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39532a05-50a1-4a28-8a9f-b2a212c757aa None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:33.018348 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 207/408] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:32 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:33.025755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51cd4f5-c73d-4fbb-87d1-fa659321c96a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:33.032583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 202/409] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:33.033997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39532a05-50a1-4a28-8a9f-b2a212c757aa None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:33.134936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:33.142447 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39532a05-50a1-4a28-8a9f-b2a212c757aa None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:33.181968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d0ce53-bca6-48df-9224-3f75530bc485 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:33.182605 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d0ce53-bca6-48df-9224-3f75530bc485 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:33.182905 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d0ce53-bca6-48df-9224-3f75530bc485 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:33.183187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d0ce53-bca6-48df-9224-3f75530bc485 None admin] PATH_INFO: `/v3/groups/4b30643760554c6fa0e17ed2e13e5aad` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:33.184305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d0ce53-bca6-48df-9224-3f75530bc485 None admin] RBAC: Authorizing `identity:get_group(group_id=4b30643760554c6fa0e17ed2e13e5aad)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:33.185751 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d0ce53-bca6-48df-9224-3f75530bc485 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:33.186566 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51cd4f5-c73d-4fbb-87d1-fa659321c96a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:33.192357 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 203/410] 10.222.0.15 () {56 vars in 1180 bytes} [Tue Sep 13 01:05:33 2022] GET /identity/v3/groups/4b30643760554c6fa0e17ed2e13e5aad => generated 229 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:33.193605 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d0ce53-bca6-48df-9224-3f75530bc485 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:33.230622 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4c4fa5-b796-47b2-b939-451adafbc3cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:33.231370 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4c4fa5-b796-47b2-b939-451adafbc3cd None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:33.231716 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4c4fa5-b796-47b2-b939-451adafbc3cd None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:33.232061 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4c4fa5-b796-47b2-b939-451adafbc3cd None admin] PATH_INFO: `/v3/groups/790da853f67443aba550f1f2b09ca790` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:33.233238 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4c4fa5-b796-47b2-b939-451adafbc3cd None admin] RBAC: Authorizing `identity:get_group(group_id=790da853f67443aba550f1f2b09ca790)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:33.234731 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4c4fa5-b796-47b2-b939-451adafbc3cd None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:33.236750 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 208/411] 10.222.0.15 () {56 vars in 1180 bytes} [Tue Sep 13 01:05:33 2022] GET /identity/v3/groups/790da853f67443aba550f1f2b09ca790 => generated 226 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:33.247882 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4c4fa5-b796-47b2-b939-451adafbc3cd None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:33.315826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d0ce53-bca6-48df-9224-3f75530bc485 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:33.332196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:33.357855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9659bda0-17ed-4494-be9e-2217ad7c210e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:33.359074 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9659bda0-17ed-4494-be9e-2217ad7c210e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:33.359503 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9659bda0-17ed-4494-be9e-2217ad7c210e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:33.359922 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9659bda0-17ed-4494-be9e-2217ad7c210e None admin] PATH_INFO: `/v3/projects/ce37ba2e4948471599c41255ecf54622` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:33.361351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9659bda0-17ed-4494-be9e-2217ad7c210e None admin] RBAC: Authorizing `identity:get_project(project_id=ce37ba2e4948471599c41255ecf54622)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:33.363118 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9659bda0-17ed-4494-be9e-2217ad7c210e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:33.365453 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 204/412] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:33 2022] GET /identity/v3/projects/ce37ba2e4948471599c41255ecf54622 => generated 344 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:33.376509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9659bda0-17ed-4494-be9e-2217ad7c210e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:33.428250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:33.513671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4c4fa5-b796-47b2-b939-451adafbc3cd None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:33.522748 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9659bda0-17ed-4494-be9e-2217ad7c210e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:33.562588 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ac180d-8508-4819-8032-49b3d4d58a74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:33.563873 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ac180d-8508-4819-8032-49b3d4d58a74 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:33.564357 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ac180d-8508-4819-8032-49b3d4d58a74 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:33.564886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ac180d-8508-4819-8032-49b3d4d58a74 None admin] PATH_INFO: `/v3/projects/ce37ba2e4948471599c41255ecf54622/groups/4b30643760554c6fa0e17ed2e13e5aad/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:33.567030 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ac180d-8508-4819-8032-49b3d4d58a74 None admin] RBAC: Authorizing `identity:create_grant(project_id=ce37ba2e4948471599c41255ecf54622, group_id=4b30643760554c6fa0e17ed2e13e5aad, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:33.569227 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ac180d-8508-4819-8032-49b3d4d58a74 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:33.596397 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052ffa89-285c-4c38-aad3-ef3203865264 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:33.597404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 205/413] 10.222.0.15 () {58 vars in 1442 bytes} [Tue Sep 13 01:05:33 2022] PUT /identity/v3/projects/ce37ba2e4948471599c41255ecf54622/groups/4b30643760554c6fa0e17ed2e13e5aad/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:05:33.598302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052ffa89-285c-4c38-aad3-ef3203865264 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:33.598993 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052ffa89-285c-4c38-aad3-ef3203865264 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:33.599473 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052ffa89-285c-4c38-aad3-ef3203865264 None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:33.600915 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052ffa89-285c-4c38-aad3-ef3203865264 None admin] RBAC: Authorizing `identity:get_project(project_id=16a6348ce8d34348b34d853f76445b3a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:33.602814 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052ffa89-285c-4c38-aad3-ef3203865264 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:33.605282 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 209/414] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:33 2022] GET /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a => generated 302 bytes in 358 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:33.611524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ac180d-8508-4819-8032-49b3d4d58a74 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:33.749634 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:33.765266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ac180d-8508-4819-8032-49b3d4d58a74 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:33.819151 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce2e2ec-1519-46ba-a77d-f78f0af10c85 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:33.819779 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce2e2ec-1519-46ba-a77d-f78f0af10c85 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:33.820278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce2e2ec-1519-46ba-a77d-f78f0af10c85 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:33.822757 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 210/415] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:33 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:05:33.828917 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcb6585-ecf3-42f2-8b44-9537d1aedfa0 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:33.829438 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a6330c-88e4-40fa-a9a3-cb9191289564 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:33.829827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcb6585-ecf3-42f2-8b44-9537d1aedfa0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:33.830118 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a6330c-88e4-40fa-a9a3-cb9191289564 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:33.830396 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a6330c-88e4-40fa-a9a3-cb9191289564 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:33.830668 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a6330c-88e4-40fa-a9a3-cb9191289564 None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a/groups/790da853f67443aba550f1f2b09ca790/roles/cd616e816a394068867091116a3f1ce0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:33.830938 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcb6585-ecf3-42f2-8b44-9537d1aedfa0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:33.836858 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a6330c-88e4-40fa-a9a3-cb9191289564 None admin] RBAC: Authorizing `identity:create_grant(project_id=16a6348ce8d34348b34d853f76445b3a, group_id=790da853f67443aba550f1f2b09ca790, role_id=cd616e816a394068867091116a3f1ce0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:33.838423 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a6330c-88e4-40fa-a9a3-cb9191289564 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:33.860025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-edcb6585-ecf3-42f2-8b44-9537d1aedfa0 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:33.861221 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-edcb6585-ecf3-42f2-8b44-9537d1aedfa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:33.861528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 206/416] 10.222.0.15 () {58 vars in 1442 bytes} [Tue Sep 13 01:05:33 2022] PUT /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a/groups/790da853f67443aba550f1f2b09ca790/roles/cd616e816a394068867091116a3f1ce0 => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:05:33.939722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 211/417] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:34.268538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a6330c-88e4-40fa-a9a3-cb9191289564 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:34.387368 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a6330c-88e4-40fa-a9a3-cb9191289564 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:34.423047 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6dbcaa-5915-4d08-bec9-5d99f27f382e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:34.423681 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6dbcaa-5915-4d08-bec9-5d99f27f382e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:34.423958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6dbcaa-5915-4d08-bec9-5d99f27f382e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:34.424231 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6dbcaa-5915-4d08-bec9-5d99f27f382e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:34.435390 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6dbcaa-5915-4d08-bec9-5d99f27f382e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:34.440959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-2a6dbcaa-5915-4d08-bec9-5d99f27f382e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:34.440959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:34.440959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:34.440959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:34.440959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:34.440959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:34.440959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:34.440959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:34.440959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:34.440959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:34.440959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:34.440959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:34.440959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:34.440959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:34.440959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:34.440959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:34.441847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:34.441847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:34.441847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:34.441847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:34.441847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:34.441847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:34.441847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:34.441847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:34.441847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:34.441847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:34.441847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:34.441847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:34.441847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:34.441847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:34.441847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:34.441847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:34.442645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:34.442645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:34.442645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:34.442645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:34.442645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:34.442645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:34.442645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:34.442645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:34.442645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:34.442645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:34.442645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:34.442645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:34.442645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 207/418] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:05:34 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:34.448800 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcb6585-ecf3-42f2-8b44-9537d1aedfa0 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:34.504965 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:34.599815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcb6585-ecf3-42f2-8b44-9537d1aedfa0 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:34.672228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e11de84-c85e-44de-a15c-5365b42602ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:34.673142 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e11de84-c85e-44de-a15c-5365b42602ef None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:34.673596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e11de84-c85e-44de-a15c-5365b42602ef None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:34.674043 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e11de84-c85e-44de-a15c-5365b42602ef None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:34.674905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e11de84-c85e-44de-a15c-5365b42602ef None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:34.675408 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e11de84-c85e-44de-a15c-5365b42602ef None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:34.676945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e11de84-c85e-44de-a15c-5365b42602ef None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:34.694004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 212/419] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:34 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:34.700963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6dbcaa-5915-4d08-bec9-5d99f27f382e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:34.719042 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:34.810580 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6dbcaa-5915-4d08-bec9-5d99f27f382e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:34.844819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfbfdc8-d2ad-4b1c-9b42-4e08d29654bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:34.845430 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfbfdc8-d2ad-4b1c-9b42-4e08d29654bd None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:34.845711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfbfdc8-d2ad-4b1c-9b42-4e08d29654bd None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:34.845983 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfbfdc8-d2ad-4b1c-9b42-4e08d29654bd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:34.918504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfbfdc8-d2ad-4b1c-9b42-4e08d29654bd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:34.919917 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfbfdc8-d2ad-4b1c-9b42-4e08d29654bd None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:34.932127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:35.032321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 208/420] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:35.033751 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e11de84-c85e-44de-a15c-5365b42602ef None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:35.166297 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:35.176112 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e11de84-c85e-44de-a15c-5365b42602ef None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:35.181677 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c1d7b6-4785-42ab-af03-ac60dccc6577 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:35.181959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c1d7b6-4785-42ab-af03-ac60dccc6577 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:35.182230 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c1d7b6-4785-42ab-af03-ac60dccc6577 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:35.185132 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 209/421] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:05:35.190041 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b68527-471d-47cf-a78c-8a235d19afde None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:35.190571 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b68527-471d-47cf-a78c-8a235d19afde None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:35.191077 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b68527-471d-47cf-a78c-8a235d19afde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:35.220131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-60b68527-471d-47cf-a78c-8a235d19afde None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:35.222622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-60b68527-471d-47cf-a78c-8a235d19afde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:35.245504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46b1177-8c93-4041-a493-5f16f1e1f8f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:35.246471 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46b1177-8c93-4041-a493-5f16f1e1f8f4 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:35.246972 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46b1177-8c93-4041-a493-5f16f1e1f8f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:35.247441 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46b1177-8c93-4041-a493-5f16f1e1f8f4 None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:35.249004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f46b1177-8c93-4041-a493-5f16f1e1f8f4 None admin] RBAC: Authorizing `identity:get_project(project_id=4426c2e7285b4cbc8331aa86690ef664)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:35.250682 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f46b1177-8c93-4041-a493-5f16f1e1f8f4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:35.252963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 213/422] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:35 2022] GET /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664 => generated 298 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:35.258433 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46b1177-8c93-4041-a493-5f16f1e1f8f4 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:35.354750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 210/423] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 01:05:35.448824 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46b1177-8c93-4041-a493-5f16f1e1f8f4 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:35.499689 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad371c66-bd7e-4077-a2ae-b4b43bdfc87a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:35.500364 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad371c66-bd7e-4077-a2ae-b4b43bdfc87a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:35.500642 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad371c66-bd7e-4077-a2ae-b4b43bdfc87a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:35.500913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad371c66-bd7e-4077-a2ae-b4b43bdfc87a None admin] PATH_INFO: `/v3/groups/790da853f67443aba550f1f2b09ca790` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:35.503568 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad371c66-bd7e-4077-a2ae-b4b43bdfc87a None admin] RBAC: Authorizing `identity:get_group(group_id=790da853f67443aba550f1f2b09ca790)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:35.505029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad371c66-bd7e-4077-a2ae-b4b43bdfc87a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:35.506803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 214/424] 10.222.0.15 () {56 vars in 1180 bytes} [Tue Sep 13 01:05:35 2022] GET /identity/v3/groups/790da853f67443aba550f1f2b09ca790 => generated 226 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:35.513376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b68527-471d-47cf-a78c-8a235d19afde None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:35.586204 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:35.662385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b68527-471d-47cf-a78c-8a235d19afde None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:35.679764 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad371c66-bd7e-4077-a2ae-b4b43bdfc87a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:35.714039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da722863-1299-4948-a022-b3f3d560c7a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:35.715972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da722863-1299-4948-a022-b3f3d560c7a3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:35.716641 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da722863-1299-4948-a022-b3f3d560c7a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:35.717244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da722863-1299-4948-a022-b3f3d560c7a3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:35.718360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da722863-1299-4948-a022-b3f3d560c7a3 None admin] RBAC: Adding query filter params (group.id=790da853f67443aba550f1f2b09ca790, role.id=cd616e816a394068867091116a3f1ce0, scope.project.id=4426c2e7285b4cbc8331aa86690ef664) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:35.719214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da722863-1299-4948-a022-b3f3d560c7a3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:35.719415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:35.721086 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da722863-1299-4948-a022-b3f3d560c7a3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:35.730096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 211/425] 10.222.0.15 () {56 vars in 1376 bytes} [Tue Sep 13 01:05:35 2022] GET /identity/v3/role_assignments?group.id=790da853f67443aba550f1f2b09ca790&scope.project.id=4426c2e7285b4cbc8331aa86690ef664&role.id=cd616e816a394068867091116a3f1ce0 => generated 611 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:35.777974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4205a0a2-6d0b-487c-a6c5-2ec0e5a3d950 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:35.778255 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4205a0a2-6d0b-487c-a6c5-2ec0e5a3d950 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:35.778526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4205a0a2-6d0b-487c-a6c5-2ec0e5a3d950 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:35.780188 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 212/426] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:05:35.785892 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d006e8-c15d-41f9-92e1-137e8d564017 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:35.786188 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d006e8-c15d-41f9-92e1-137e8d564017 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:35.786451 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d006e8-c15d-41f9-92e1-137e8d564017 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:35.807371 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d3d006e8-c15d-41f9-92e1-137e8d564017 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:35.807605 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d3d006e8-c15d-41f9-92e1-137e8d564017 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:35.841821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad371c66-bd7e-4077-a2ae-b4b43bdfc87a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:35.882603 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff3d595-97b3-47d6-9c9d-27a8fc061f72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:35.883332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff3d595-97b3-47d6-9c9d-27a8fc061f72 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:35.883615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff3d595-97b3-47d6-9c9d-27a8fc061f72 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:35.883889 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff3d595-97b3-47d6-9c9d-27a8fc061f72 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:35.902836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff3d595-97b3-47d6-9c9d-27a8fc061f72 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:35.908563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-bff3d595-97b3-47d6-9c9d-27a8fc061f72 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:35.908563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:35.908563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:35.908563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:35.908563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:35.908563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:35.908563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:35.908563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:35.908563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:35.908563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:35.908563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:35.908563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:35.908563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:35.908563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:35.908563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:35.908563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:35.909504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:35.909504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:35.909504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:35.909504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:35.909504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:35.909504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:35.909504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:35.909504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:35.909504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:35.909504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:35.909504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:35.909504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:35.909504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:35.909504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:35.909504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:35.909504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:35.910283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:35.910283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:35.910283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:35.910283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:35.910283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:35.910283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:35.910283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:35.910283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:35.910283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:35.910283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:35.910283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:35.910283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:35.910283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 215/427] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:35 2022] GET /identity/v3/roles/member => generated 84 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:35.915812 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff3d595-97b3-47d6-9c9d-27a8fc061f72 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:35.946299 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 213/428] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:36.013755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:36.028472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff3d595-97b3-47d6-9c9d-27a8fc061f72 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:36.074597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c18bc7e-d4a6-403b-b1c3-96aaec64776e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:36.075363 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c18bc7e-d4a6-403b-b1c3-96aaec64776e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:36.075796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c18bc7e-d4a6-403b-b1c3-96aaec64776e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:36.076218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c18bc7e-d4a6-403b-b1c3-96aaec64776e None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:36.077240 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c18bc7e-d4a6-403b-b1c3-96aaec64776e None admin] RBAC: Adding query filter params (name=member) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:36.077720 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c18bc7e-d4a6-403b-b1c3-96aaec64776e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:36.079935 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c18bc7e-d4a6-403b-b1c3-96aaec64776e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:36.093075 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 216/429] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:35 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:36.100648 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d006e8-c15d-41f9-92e1-137e8d564017 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:36.136558 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:36.206098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d006e8-c15d-41f9-92e1-137e8d564017 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:36.241168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0874c5-32b0-4790-8cc5-bc334b83b24e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:36.241821 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0874c5-32b0-4790-8cc5-bc334b83b24e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:36.242130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0874c5-32b0-4790-8cc5-bc334b83b24e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:36.242439 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0874c5-32b0-4790-8cc5-bc334b83b24e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:36.245811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c18bc7e-d4a6-403b-b1c3-96aaec64776e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:36.319806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0874c5-32b0-4790-8cc5-bc334b83b24e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:36.321457 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0874c5-32b0-4790-8cc5-bc334b83b24e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:36.355334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:36.364676 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c18bc7e-d4a6-403b-b1c3-96aaec64776e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:36.423895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e65f17e-bd1b-4fb1-8b4a-c9c94fed3feb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:36.424771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e65f17e-bd1b-4fb1-8b4a-c9c94fed3feb None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:36.425109 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e65f17e-bd1b-4fb1-8b4a-c9c94fed3feb None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:36.425388 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e65f17e-bd1b-4fb1-8b4a-c9c94fed3feb None admin] PATH_INFO: `/v3/roles/member` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:36.434717 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 214/430] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:36.434717 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dda75ee-b1bf-493f-aa25-249331bde061 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:36.434717 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dda75ee-b1bf-493f-aa25-249331bde061 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:36.434717 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dda75ee-b1bf-493f-aa25-249331bde061 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:36.434717 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 215/431] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:36.436451 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dda75ee-b1bf-493f-aa25-249331bde061 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:36.443226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e65f17e-bd1b-4fb1-8b4a-c9c94fed3feb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:36.449089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-4e65f17e-bd1b-4fb1-8b4a-c9c94fed3feb None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:36.449089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:36.449089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:36.449089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:36.449089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:36.449089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:36.449089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:36.449089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:36.449089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:36.449089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:36.449089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:36.449089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:36.449089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:36.449089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:36.449089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:36.449089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:36.450365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:36.450365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:36.450365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:36.450365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:36.450365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:36.450365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:36.450365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:36.450365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:36.450365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:36.450365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:36.450365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:36.450365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:36.450365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:36.450365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:36.450365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:36.450365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:36.451712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:36.451712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:36.451712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:36.451712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:36.451712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:36.451712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:36.451712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:36.451712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:36.451712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:36.451712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:36.451712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 01:05:36.451712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:36.451712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 217/432] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:36 2022] GET /identity/v3/roles/member => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:36.451712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6ec7e8-882e-4882-ae8a-bcd07fcac0ea None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:36.452410 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6ec7e8-882e-4882-ae8a-bcd07fcac0ea None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:36.452410 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6ec7e8-882e-4882-ae8a-bcd07fcac0ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:36.471830 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-1e6ec7e8-882e-4882-ae8a-bcd07fcac0ea None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:36.473073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1e6ec7e8-882e-4882-ae8a-bcd07fcac0ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:36.476971 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:36.544876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 218/433] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:36.545999 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6ec7e8-882e-4882-ae8a-bcd07fcac0ea None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:36.566141 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dda75ee-b1bf-493f-aa25-249331bde061 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:36.615219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f88d322-d37d-4072-9871-981984f75701 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:36.616155 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f88d322-d37d-4072-9871-981984f75701 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:36.616698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f88d322-d37d-4072-9871-981984f75701 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:36.616975 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f88d322-d37d-4072-9871-981984f75701 None admin] PATH_INFO: `/v3/projects/4426c2e7285b4cbc8331aa86690ef664` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:36.618118 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f88d322-d37d-4072-9871-981984f75701 None admin] RBAC: Authorizing `identity:get_project(project_id=4426c2e7285b4cbc8331aa86690ef664)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:36.620377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f88d322-d37d-4072-9871-981984f75701 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:36.622418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 216/434] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:36 2022] GET /identity/v3/projects/4426c2e7285b4cbc8331aa86690ef664 => generated 298 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:36.627914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c305ce4f-42f3-4e95-8dd6-99b09996a50c None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:36.628354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c305ce4f-42f3-4e95-8dd6-99b09996a50c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:36.628626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c305ce4f-42f3-4e95-8dd6-99b09996a50c None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:36.630005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 217/435] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:36 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:36.632014 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c305ce4f-42f3-4e95-8dd6-99b09996a50c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:36.636203 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:36.654388 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6ec7e8-882e-4882-ae8a-bcd07fcac0ea None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:36.687771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61ebcb1-95f6-4169-8797-3e7917d0ae4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:36.688634 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61ebcb1-95f6-4169-8797-3e7917d0ae4d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:36.688911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61ebcb1-95f6-4169-8797-3e7917d0ae4d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:36.689179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61ebcb1-95f6-4169-8797-3e7917d0ae4d None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:36.689845 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61ebcb1-95f6-4169-8797-3e7917d0ae4d None admin] RBAC: Adding query filter params (name=member) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:36.690145 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61ebcb1-95f6-4169-8797-3e7917d0ae4d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:36.691597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61ebcb1-95f6-4169-8797-3e7917d0ae4d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:36.699901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 219/436] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:05:36 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:36.706199 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d80b49a-8db7-456f-b488-5d67c707e6dd None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:36.706482 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d80b49a-8db7-456f-b488-5d67c707e6dd None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:36.706751 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d80b49a-8db7-456f-b488-5d67c707e6dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:36.714376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:36.727318 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-2d80b49a-8db7-456f-b488-5d67c707e6dd None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:36.728491 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2d80b49a-8db7-456f-b488-5d67c707e6dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:36.744867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c305ce4f-42f3-4e95-8dd6-99b09996a50c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:36.781629 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0ce706-0afb-4622-80aa-a6922142a1c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:36.782477 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0ce706-0afb-4622-80aa-a6922142a1c2 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:36.782926 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0ce706-0afb-4622-80aa-a6922142a1c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:36.783334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0ce706-0afb-4622-80aa-a6922142a1c2 None admin] PATH_INFO: `/v3/groups/790da853f67443aba550f1f2b09ca790` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:36.784564 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0ce706-0afb-4622-80aa-a6922142a1c2 None admin] RBAC: Authorizing `identity:get_group(group_id=790da853f67443aba550f1f2b09ca790)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:36.786120 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0ce706-0afb-4622-80aa-a6922142a1c2 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:36.792775 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 218/437] 10.222.0.15 () {56 vars in 1180 bytes} [Tue Sep 13 01:05:36 2022] GET /identity/v3/groups/790da853f67443aba550f1f2b09ca790 => generated 226 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:36.793744 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0ce706-0afb-4622-80aa-a6922142a1c2 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:36.798335 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 220/438] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:36.799188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d80b49a-8db7-456f-b488-5d67c707e6dd None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:36.869055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:36.904666 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0ce706-0afb-4622-80aa-a6922142a1c2 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:36.909511 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d80b49a-8db7-456f-b488-5d67c707e6dd None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:36.937622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c068d74-1a56-4d8a-b426-e6d15670fdd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:36.938498 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c068d74-1a56-4d8a-b426-e6d15670fdd6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:36.938776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c068d74-1a56-4d8a-b426-e6d15670fdd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:36.939078 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c068d74-1a56-4d8a-b426-e6d15670fdd6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:36.942287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:36.948439 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c05bb37-6bb0-4339-bcb7-622ceed20116 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:36.949251 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c05bb37-6bb0-4339-bcb7-622ceed20116 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:36.949703 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c05bb37-6bb0-4339-bcb7-622ceed20116 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:36.950148 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c05bb37-6bb0-4339-bcb7-622ceed20116 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:36.951165 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c05bb37-6bb0-4339-bcb7-622ceed20116 None admin] RBAC: Adding query filter params (group.id=790da853f67443aba550f1f2b09ca790, role.id=6f71034643cb4e8caee299f5d0bb863f, scope.project.id=4426c2e7285b4cbc8331aa86690ef664) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:36.951663 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c05bb37-6bb0-4339-bcb7-622ceed20116 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:36.953256 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c05bb37-6bb0-4339-bcb7-622ceed20116 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:36.959564 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 221/439] 10.222.0.15 () {56 vars in 1376 bytes} [Tue Sep 13 01:05:36 2022] GET /identity/v3/role_assignments?group.id=790da853f67443aba550f1f2b09ca790&scope.project.id=4426c2e7285b4cbc8331aa86690ef664&role.id=6f71034643cb4e8caee299f5d0bb863f => generated 611 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:36.960532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c05bb37-6bb0-4339-bcb7-622ceed20116 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:37.015720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c068d74-1a56-4d8a-b426-e6d15670fdd6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:37.017365 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c068d74-1a56-4d8a-b426-e6d15670fdd6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:37.100155 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:37.106383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c05bb37-6bb0-4339-bcb7-622ceed20116 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:37.141909 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 219/440] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:37.141909 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c068d74-1a56-4d8a-b426-e6d15670fdd6 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:37.150007 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff9ce20-69d9-408c-b110-bd48d6415fce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:37.151191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff9ce20-69d9-408c-b110-bd48d6415fce None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:37.151191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff9ce20-69d9-408c-b110-bd48d6415fce None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:37.151191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff9ce20-69d9-408c-b110-bd48d6415fce None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:37.165662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff9ce20-69d9-408c-b110-bd48d6415fce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:37.170896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-bff9ce20-69d9-408c-b110-bd48d6415fce None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:05:37.170896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:37.170896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:37.170896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:37.170896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:37.170896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:37.170896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:37.170896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:37.170896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:37.170896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:37.170896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:37.170896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:37.170896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:37.170896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:37.170896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:37.170896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:37.171766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:37.171766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:37.171766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:37.171766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:37.171766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:37.171766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:37.171766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:37.171766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:37.171766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:37.171766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:37.171766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:37.171766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:37.171766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:37.171766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:37.171766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:37.171766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:37.172570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:37.172570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:37.172570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:37.172570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:37.172570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:37.172570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:37.172570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:37.172570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:37.172570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:37.172570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:37.172570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:05:37.172570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:37.172570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 222/441] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:05:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:37.173207 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff9ce20-69d9-408c-b110-bd48d6415fce None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:37.239411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:37.254144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c068d74-1a56-4d8a-b426-e6d15670fdd6 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:37.290481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8214f0e1-f7a1-4e83-8c21-092a6dc41266 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:37.291134 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8214f0e1-f7a1-4e83-8c21-092a6dc41266 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:37.291437 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8214f0e1-f7a1-4e83-8c21-092a6dc41266 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:37.291437 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8214f0e1-f7a1-4e83-8c21-092a6dc41266 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:37.298568 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff9ce20-69d9-408c-b110-bd48d6415fce None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:37.303477 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8214f0e1-f7a1-4e83-8c21-092a6dc41266 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:37.308914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-8214f0e1-f7a1-4e83-8c21-092a6dc41266 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:37.308914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:37.308914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:37.308914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:37.308914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:37.308914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:37.308914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:37.308914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:37.308914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:37.308914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:37.308914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:37.308914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:37.308914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:37.308914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:37.308914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:37.308914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:37.309829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:37.309829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:37.309829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:37.309829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:37.309829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:37.309829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:37.309829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:37.309829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:37.309829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:37.309829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:37.309829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:37.309829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:37.309829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:37.309829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:37.309829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:37.309829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:37.310622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:37.310622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:37.310622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:37.310622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:37.310622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:37.310622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:37.310622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:37.310622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:37.310622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:37.310622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:37.310622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:37.310622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:37.310622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 220/442] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:05:37 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:37.311308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8214f0e1-f7a1-4e83-8c21-092a6dc41266 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:37.337146 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6638cece-3b0f-4b54-a92f-36a25b60a110 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:37.338019 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6638cece-3b0f-4b54-a92f-36a25b60a110 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:37.338441 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6638cece-3b0f-4b54-a92f-36a25b60a110 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:37.338905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6638cece-3b0f-4b54-a92f-36a25b60a110 None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:37.340184 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6638cece-3b0f-4b54-a92f-36a25b60a110 None admin] RBAC: Authorizing `identity:get_project(project_id=16a6348ce8d34348b34d853f76445b3a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:37.341649 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6638cece-3b0f-4b54-a92f-36a25b60a110 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:37.343296 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 223/443] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:37 2022] GET /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a => generated 302 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:37.344062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6638cece-3b0f-4b54-a92f-36a25b60a110 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:37.347156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:37.426921 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8214f0e1-f7a1-4e83-8c21-092a6dc41266 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:37.469913 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab868311-71b7-4acc-923e-4609fc979e56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:37.470527 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab868311-71b7-4acc-923e-4609fc979e56 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:37.470803 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab868311-71b7-4acc-923e-4609fc979e56 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:37.471098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab868311-71b7-4acc-923e-4609fc979e56 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:37.471768 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab868311-71b7-4acc-923e-4609fc979e56 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:37.473110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab868311-71b7-4acc-923e-4609fc979e56 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:37.473581 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab868311-71b7-4acc-923e-4609fc979e56 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:37.482026 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6638cece-3b0f-4b54-a92f-36a25b60a110 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:37.483637 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 221/444] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:37.484458 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab868311-71b7-4acc-923e-4609fc979e56 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:37.510897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:37.521688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e008cbe1-64c7-4ceb-badc-dfbdf6f18ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:37.522302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e008cbe1-64c7-4ceb-badc-dfbdf6f18ce7 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:37.522578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e008cbe1-64c7-4ceb-badc-dfbdf6f18ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:37.522844 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e008cbe1-64c7-4ceb-badc-dfbdf6f18ce7 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:37.524057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e008cbe1-64c7-4ceb-badc-dfbdf6f18ce7 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:37.525222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e008cbe1-64c7-4ceb-badc-dfbdf6f18ce7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:37.525710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e008cbe1-64c7-4ceb-badc-dfbdf6f18ce7 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:37.533825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 224/445] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:37 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:37.534621 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e008cbe1-64c7-4ceb-badc-dfbdf6f18ce7 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:37.580103 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:37.589813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab868311-71b7-4acc-923e-4609fc979e56 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:37.624393 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a283a63-edbb-4a55-8e63-fb8ccae6b14c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:37.625019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a283a63-edbb-4a55-8e63-fb8ccae6b14c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:37.625307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a283a63-edbb-4a55-8e63-fb8ccae6b14c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:37.625576 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a283a63-edbb-4a55-8e63-fb8ccae6b14c None admin] PATH_INFO: `/v3/groups/790da853f67443aba550f1f2b09ca790` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:37.626691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a283a63-edbb-4a55-8e63-fb8ccae6b14c None admin] RBAC: Authorizing `identity:get_group(group_id=790da853f67443aba550f1f2b09ca790)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:37.628162 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a283a63-edbb-4a55-8e63-fb8ccae6b14c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:37.629727 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 222/446] 10.222.0.15 () {56 vars in 1180 bytes} [Tue Sep 13 01:05:37 2022] GET /identity/v3/groups/790da853f67443aba550f1f2b09ca790 => generated 226 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:37.630478 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a283a63-edbb-4a55-8e63-fb8ccae6b14c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:37.634698 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e008cbe1-64c7-4ceb-badc-dfbdf6f18ce7 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:37.668287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085b58e1-1d24-43ab-ad55-497aa8862321 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:37.668826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085b58e1-1d24-43ab-ad55-497aa8862321 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:37.669100 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085b58e1-1d24-43ab-ad55-497aa8862321 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:37.669369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085b58e1-1d24-43ab-ad55-497aa8862321 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:37.723833 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:37.728322 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a283a63-edbb-4a55-8e63-fb8ccae6b14c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:37.738083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085b58e1-1d24-43ab-ad55-497aa8862321 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:37.739550 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085b58e1-1d24-43ab-ad55-497aa8862321 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:37.761651 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f7e8ab-9a3d-4ab5-9199-85ae7b193d31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:37.762248 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f7e8ab-9a3d-4ab5-9199-85ae7b193d31 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:37.762519 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f7e8ab-9a3d-4ab5-9199-85ae7b193d31 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:37.762784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f7e8ab-9a3d-4ab5-9199-85ae7b193d31 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:37.789483 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:37.828645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f7e8ab-9a3d-4ab5-9199-85ae7b193d31 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:37.830103 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f7e8ab-9a3d-4ab5-9199-85ae7b193d31 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:37.835828 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 225/447] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:37.836651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085b58e1-1d24-43ab-ad55-497aa8862321 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:37.918739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:37.931284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 223/448] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:37.932107 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f7e8ab-9a3d-4ab5-9199-85ae7b193d31 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:37.934337 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085b58e1-1d24-43ab-ad55-497aa8862321 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:37.967338 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc3e41c-939f-4937-be56-5d5da6e889cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:37.967932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc3e41c-939f-4937-be56-5d5da6e889cc None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:37.968205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc3e41c-939f-4937-be56-5d5da6e889cc None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:37.968472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc3e41c-939f-4937-be56-5d5da6e889cc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:37.969180 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc3e41c-939f-4937-be56-5d5da6e889cc None admin] RBAC: Adding query filter params (group.id=790da853f67443aba550f1f2b09ca790, role.id=6f71034643cb4e8caee299f5d0bb863f, scope.project.id=16a6348ce8d34348b34d853f76445b3a) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:37.969482 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc3e41c-939f-4937-be56-5d5da6e889cc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:37.970905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc3e41c-939f-4937-be56-5d5da6e889cc None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:37.976428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 226/449] 10.222.0.15 () {56 vars in 1376 bytes} [Tue Sep 13 01:05:37 2022] GET /identity/v3/role_assignments?group.id=790da853f67443aba550f1f2b09ca790&scope.project.id=16a6348ce8d34348b34d853f76445b3a&role.id=6f71034643cb4e8caee299f5d0bb863f => generated 611 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:37.977207 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc3e41c-939f-4937-be56-5d5da6e889cc None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:37.991299 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:38.043940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f7e8ab-9a3d-4ab5-9199-85ae7b193d31 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:38.081097 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47a1253-993b-4ef2-acc6-8474b531010d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:38.081744 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47a1253-993b-4ef2-acc6-8474b531010d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:38.082022 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47a1253-993b-4ef2-acc6-8474b531010d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:38.082299 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47a1253-993b-4ef2-acc6-8474b531010d None admin] PATH_INFO: `/v3/projects/ce37ba2e4948471599c41255ecf54622` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:38.083477 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47a1253-993b-4ef2-acc6-8474b531010d None admin] RBAC: Authorizing `identity:get_project(project_id=ce37ba2e4948471599c41255ecf54622)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:38.084937 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47a1253-993b-4ef2-acc6-8474b531010d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:38.087190 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 224/450] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:37 2022] GET /identity/v3/projects/ce37ba2e4948471599c41255ecf54622 => generated 344 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:38.092778 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47a1253-993b-4ef2-acc6-8474b531010d None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:38.110404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc3e41c-939f-4937-be56-5d5da6e889cc None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:38.149735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6090301-532d-41fa-b802-0b5f0200b3d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:38.150364 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6090301-532d-41fa-b802-0b5f0200b3d8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:38.150647 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6090301-532d-41fa-b802-0b5f0200b3d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:38.150942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6090301-532d-41fa-b802-0b5f0200b3d8 None admin] PATH_INFO: `/v3/projects/16a6348ce8d34348b34d853f76445b3a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:38.152069 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6090301-532d-41fa-b802-0b5f0200b3d8 None admin] RBAC: Authorizing `identity:get_project(project_id=16a6348ce8d34348b34d853f76445b3a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:38.153510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6090301-532d-41fa-b802-0b5f0200b3d8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:38.155472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 227/451] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:05:37 2022] GET /identity/v3/projects/16a6348ce8d34348b34d853f76445b3a => generated 302 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:38.160844 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6090301-532d-41fa-b802-0b5f0200b3d8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:38.184522 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:38.212404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47a1253-993b-4ef2-acc6-8474b531010d None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:38.247107 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:38.257769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec72f7d6-76c8-463d-a973-a0119b5167ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:38.258435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec72f7d6-76c8-463d-a973-a0119b5167ec None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:38.258674 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec72f7d6-76c8-463d-a973-a0119b5167ec None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:38.261358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec72f7d6-76c8-463d-a973-a0119b5167ec None admin] PATH_INFO: `/v3/groups/4b30643760554c6fa0e17ed2e13e5aad` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:38.262125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec72f7d6-76c8-463d-a973-a0119b5167ec None admin] RBAC: Authorizing `identity:get_group(group_id=4b30643760554c6fa0e17ed2e13e5aad)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:38.263427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec72f7d6-76c8-463d-a973-a0119b5167ec None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:38.264982 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 225/452] 10.222.0.15 () {56 vars in 1180 bytes} [Tue Sep 13 01:05:38 2022] GET /identity/v3/groups/4b30643760554c6fa0e17ed2e13e5aad => generated 229 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:38.271013 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec72f7d6-76c8-463d-a973-a0119b5167ec None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:38.283488 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6090301-532d-41fa-b802-0b5f0200b3d8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:38.330805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d046c98-e0e0-42e6-b200-55bc339e081f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:38.331468 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d046c98-e0e0-42e6-b200-55bc339e081f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:38.331749 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d046c98-e0e0-42e6-b200-55bc339e081f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:38.332028 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d046c98-e0e0-42e6-b200-55bc339e081f None admin] PATH_INFO: `/v3/groups/790da853f67443aba550f1f2b09ca790` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:38.334008 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d046c98-e0e0-42e6-b200-55bc339e081f None admin] RBAC: Authorizing `identity:get_group(group_id=790da853f67443aba550f1f2b09ca790)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:38.335557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d046c98-e0e0-42e6-b200-55bc339e081f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:38.337705 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 228/453] 10.222.0.15 () {56 vars in 1180 bytes} [Tue Sep 13 01:05:38 2022] GET /identity/v3/groups/790da853f67443aba550f1f2b09ca790 => generated 226 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:38.344676 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d046c98-e0e0-42e6-b200-55bc339e081f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:38.381097 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec72f7d6-76c8-463d-a973-a0119b5167ec None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:38.414829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1c542e-8b78-4929-9b8d-ced791b3e445 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:38.415453 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1c542e-8b78-4929-9b8d-ced791b3e445 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:38.415727 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1c542e-8b78-4929-9b8d-ced791b3e445 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:38.415998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1c542e-8b78-4929-9b8d-ced791b3e445 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:38.416706 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1c542e-8b78-4929-9b8d-ced791b3e445 None admin] RBAC: Adding query filter params (group.id=4b30643760554c6fa0e17ed2e13e5aad, role.id=66a36c858aab40688080eed6ceace3a0, scope.project.id=ce37ba2e4948471599c41255ecf54622) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:38.417006 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1c542e-8b78-4929-9b8d-ced791b3e445 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:38.418435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1c542e-8b78-4929-9b8d-ced791b3e445 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:38.424107 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 226/454] 10.222.0.15 () {56 vars in 1376 bytes} [Tue Sep 13 01:05:38 2022] GET /identity/v3/role_assignments?group.id=4b30643760554c6fa0e17ed2e13e5aad&scope.project.id=ce37ba2e4948471599c41255ecf54622&role.id=66a36c858aab40688080eed6ceace3a0 => generated 611 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:38.438672 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:38.454815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d046c98-e0e0-42e6-b200-55bc339e081f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:38.490505 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8154cc2-5151-46a9-82a4-2a6bdcb26243 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:38.491237 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8154cc2-5151-46a9-82a4-2a6bdcb26243 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:38.491411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8154cc2-5151-46a9-82a4-2a6bdcb26243 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:38.491687 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8154cc2-5151-46a9-82a4-2a6bdcb26243 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:38.493043 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8154cc2-5151-46a9-82a4-2a6bdcb26243 None admin] RBAC: Adding query filter params (group.id=790da853f67443aba550f1f2b09ca790, role.id=cd616e816a394068867091116a3f1ce0, scope.project.id=16a6348ce8d34348b34d853f76445b3a) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:38.493349 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8154cc2-5151-46a9-82a4-2a6bdcb26243 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:38.494796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8154cc2-5151-46a9-82a4-2a6bdcb26243 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:38.501415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 229/455] 10.222.0.15 () {56 vars in 1376 bytes} [Tue Sep 13 01:05:38 2022] GET /identity/v3/role_assignments?group.id=790da853f67443aba550f1f2b09ca790&scope.project.id=16a6348ce8d34348b34d853f76445b3a&role.id=cd616e816a394068867091116a3f1ce0 => generated 611 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:41.208371 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f9d8f3-c14d-4648-b46f-c378f3991527 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:41.209368 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f9d8f3-c14d-4648-b46f-c378f3991527 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:41.209852 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f9d8f3-c14d-4648-b46f-c378f3991527 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:41.212321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 227/456] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:41 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:41.218520 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c25a0a-4dfe-4ee5-bd03-c1c5c7fec213 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:41.218992 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c25a0a-4dfe-4ee5-bd03-c1c5c7fec213 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:41.219065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c25a0a-4dfe-4ee5-bd03-c1c5c7fec213 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:41.243057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a3c25a0a-4dfe-4ee5-bd03-c1c5c7fec213 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:41.244219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a3c25a0a-4dfe-4ee5-bd03-c1c5c7fec213 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:41.341867 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 230/457] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:41.488870 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03fff8b-cb55-4da9-8fb8-5602fec8b00c None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:41.489506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03fff8b-cb55-4da9-8fb8-5602fec8b00c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:41.489978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03fff8b-cb55-4da9-8fb8-5602fec8b00c None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:41.491807 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 228/458] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:41.496899 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210e284e-548b-49a5-a7e1-03c4415d554c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:41.497184 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210e284e-548b-49a5-a7e1-03c4415d554c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:41.497455 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210e284e-548b-49a5-a7e1-03c4415d554c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:41.519851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-210e284e-548b-49a5-a7e1-03c4415d554c None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:41.521030 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-210e284e-548b-49a5-a7e1-03c4415d554c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:41.601900 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 231/459] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:41.659873 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03fff8b-cb55-4da9-8fb8-5602fec8b00c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:41.743986 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:41.793495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b7ef03-c498-4477-8939-697cf498795a None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:41.793803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b7ef03-c498-4477-8939-697cf498795a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:41.794046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b7ef03-c498-4477-8939-697cf498795a None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:41.795694 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 232/460] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:05:41.801376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9730cf5e-74d2-425a-bfbc-509142bbaff7 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:41.801660 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9730cf5e-74d2-425a-bfbc-509142bbaff7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:41.801939 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9730cf5e-74d2-425a-bfbc-509142bbaff7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:41.805539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03fff8b-cb55-4da9-8fb8-5602fec8b00c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:41.821476 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9730cf5e-74d2-425a-bfbc-509142bbaff7 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:41.822666 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9730cf5e-74d2-425a-bfbc-509142bbaff7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:41.826080 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:41.880449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9871a653-a5c2-4d4a-b70a-bb9aec23e9f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:41.881505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9871a653-a5c2-4d4a-b70a-bb9aec23e9f1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:41.881997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9871a653-a5c2-4d4a-b70a-bb9aec23e9f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:41.882463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9871a653-a5c2-4d4a-b70a-bb9aec23e9f1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:41.898877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 233/461] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 01:05:41.908037 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9871a653-a5c2-4d4a-b70a-bb9aec23e9f1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:41.909791 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9871a653-a5c2-4d4a-b70a-bb9aec23e9f1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:41.935212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-9871a653-a5c2-4d4a-b70a-bb9aec23e9f1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:41.935212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:41.935212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:41.935212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:41.935212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:41.935212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:41.935212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:41.935212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:41.935212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:41.935212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:41.935212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:41.935212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:41.935212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:41.935212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:41.935212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:41.935212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:41.936147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:41.936147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:41.936147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:41.936147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:41.936147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:41.936147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:41.936147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:41.936147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:41.936147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:41.936147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:41.936147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:41.936147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:41.936147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:41.936147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:41.936147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:41.936147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:41.936147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:41.936985 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:41.936985 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:41.936985 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:41.939323 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 229/462] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:41.945592 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9730cf5e-74d2-425a-bfbc-509142bbaff7 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:41.965569 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9871a653-a5c2-4d4a-b70a-bb9aec23e9f1 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:42.064003 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9730cf5e-74d2-425a-bfbc-509142bbaff7 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:42.101201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c25a0ff-d408-4383-b282-837aeda555c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:42.102335 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c25a0ff-d408-4383-b282-837aeda555c9 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:42.102611 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c25a0ff-d408-4383-b282-837aeda555c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:42.102894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c25a0ff-d408-4383-b282-837aeda555c9 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:42.103535 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c25a0ff-d408-4383-b282-837aeda555c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:42.103835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c25a0ff-d408-4383-b282-837aeda555c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:42.109964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c25a0ff-d408-4383-b282-837aeda555c9 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:42.115757 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:42.119211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 234/463] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:42.120519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4492f1a-e86c-4062-a692-568ff089aa80 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:42.121138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4492f1a-e86c-4062-a692-568ff089aa80 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:42.121720 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4492f1a-e86c-4062-a692-568ff089aa80 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:42.126898 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 235/464] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:42.126898 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4492f1a-e86c-4062-a692-568ff089aa80 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:42.151638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9871a653-a5c2-4d4a-b70a-bb9aec23e9f1 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:42.165555 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:42.188388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0473cbce-cbc8-4bf8-9173-829a8b9a9fef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:42.189316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0473cbce-cbc8-4bf8-9173-829a8b9a9fef None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:42.189813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0473cbce-cbc8-4bf8-9173-829a8b9a9fef None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:42.190300 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0473cbce-cbc8-4bf8-9173-829a8b9a9fef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:42.201824 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0473cbce-cbc8-4bf8-9173-829a8b9a9fef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:42.203539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0473cbce-cbc8-4bf8-9173-829a8b9a9fef None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:42.218691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-0473cbce-cbc8-4bf8-9173-829a8b9a9fef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:42.218691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:42.218691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:42.218691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:42.218691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:42.218691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:42.218691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:42.218691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:42.218691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:42.218691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:42.218691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:42.218691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:42.218691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:42.218691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:42.218691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:42.218691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:42.220046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:42.220046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:42.220046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:42.220046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:42.220046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:42.220046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:42.220046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:42.220046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:42.220046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:42.220046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:42.220046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:42.220046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:42.220046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:42.220046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:42.220046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:42.220046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:42.220046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:42.220889 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:42.220889 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:42.220889 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:42.222683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 230/465] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:05:42.224084 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67944592-d47b-42cd-a534-3b613f1e3f21 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:42.224595 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67944592-d47b-42cd-a534-3b613f1e3f21 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:42.225068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67944592-d47b-42cd-a534-3b613f1e3f21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:42.234465 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4492f1a-e86c-4062-a692-568ff089aa80 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:42.245598 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-67944592-d47b-42cd-a534-3b613f1e3f21 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:42.247084 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-67944592-d47b-42cd-a534-3b613f1e3f21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:42.270417 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dbba92-e98f-445d-a600-4babee3c2958 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:42.271171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dbba92-e98f-445d-a600-4babee3c2958 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:42.271516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dbba92-e98f-445d-a600-4babee3c2958 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:42.271794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dbba92-e98f-445d-a600-4babee3c2958 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:42.272453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58dbba92-e98f-445d-a600-4babee3c2958 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:42.273870 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58dbba92-e98f-445d-a600-4babee3c2958 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:42.298905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 236/466] 10.222.0.15 () {60 vars in 1132 bytes} [Tue Sep 13 01:05:42 2022] POST /identity/v3/users => generated 246 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:05:42.298905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dbba92-e98f-445d-a600-4babee3c2958 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:42.342714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 231/467] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:42.342714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67944592-d47b-42cd-a534-3b613f1e3f21 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:42.388944 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:42.473979 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dbba92-e98f-445d-a600-4babee3c2958 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:42.520008 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6783efff-686f-491a-8c70-bdbad07c6665 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:42.520626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6783efff-686f-491a-8c70-bdbad07c6665 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:42.520968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6783efff-686f-491a-8c70-bdbad07c6665 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:42.521793 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6783efff-686f-491a-8c70-bdbad07c6665 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:42.522589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6783efff-686f-491a-8c70-bdbad07c6665 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:42.523013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6783efff-686f-491a-8c70-bdbad07c6665 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:42.525038 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6783efff-686f-491a-8c70-bdbad07c6665 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:42.528579 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:42.536362 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67944592-d47b-42cd-a534-3b613f1e3f21 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:42.542333 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 237/468] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:42.547663 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6783efff-686f-491a-8c70-bdbad07c6665 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:42.591639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef087e86-1eca-462a-ad3b-9a8e96e80c35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:42.592885 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef087e86-1eca-462a-ad3b-9a8e96e80c35 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:42.593110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef087e86-1eca-462a-ad3b-9a8e96e80c35 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:42.593386 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef087e86-1eca-462a-ad3b-9a8e96e80c35 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:42.609138 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef087e86-1eca-462a-ad3b-9a8e96e80c35 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:42.611349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef087e86-1eca-462a-ad3b-9a8e96e80c35 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:42.631250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-ef087e86-1eca-462a-ad3b-9a8e96e80c35 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:42.631250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:42.631250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:42.631250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:42.631250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:42.631250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:42.631250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:42.631250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:42.631250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:42.631250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:42.631250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:42.631250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:42.631250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:42.631250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:42.631250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:42.631250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:42.632156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:42.632156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:42.632156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:42.632156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:42.632156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:42.632156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:42.632156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:42.632156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:42.632156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:42.632156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:42.632156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:42.632156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:42.632156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:42.632156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:42.632156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:42.632156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:42.632156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:42.633452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:42.633452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:42.633452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:42.633452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 232/469] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:42.638783 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef087e86-1eca-462a-ad3b-9a8e96e80c35 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:42.658325 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6783efff-686f-491a-8c70-bdbad07c6665 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:42.676296 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:42.700949 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7fac70-bd10-46fd-84d7-b750f404cf2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:42.701558 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7fac70-bd10-46fd-84d7-b750f404cf2b None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:42.701838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7fac70-bd10-46fd-84d7-b750f404cf2b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:42.702105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7fac70-bd10-46fd-84d7-b750f404cf2b None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:42.702753 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7fac70-bd10-46fd-84d7-b750f404cf2b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:42.704885 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7fac70-bd10-46fd-84d7-b750f404cf2b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:42.736801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 238/470] 10.222.0.15 () {60 vars in 1132 bytes} [Tue Sep 13 01:05:42 2022] POST /identity/v3/users => generated 248 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:05:42.737807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7fac70-bd10-46fd-84d7-b750f404cf2b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:42.773058 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef087e86-1eca-462a-ad3b-9a8e96e80c35 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:42.830081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2683dc68-93ac-455b-8bcf-feb9ef976a26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:42.831191 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2683dc68-93ac-455b-8bcf-feb9ef976a26 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:42.832001 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2683dc68-93ac-455b-8bcf-feb9ef976a26 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:42.833546 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2683dc68-93ac-455b-8bcf-feb9ef976a26 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:42.834453 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2683dc68-93ac-455b-8bcf-feb9ef976a26 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:42.835199 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2683dc68-93ac-455b-8bcf-feb9ef976a26 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:42.837788 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2683dc68-93ac-455b-8bcf-feb9ef976a26 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:42.860811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 233/471] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:42.862660 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2683dc68-93ac-455b-8bcf-feb9ef976a26 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:42.873188 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:42.882013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7fac70-bd10-46fd-84d7-b750f404cf2b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:42.932248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c335b5-fe29-4690-afc8-391c049bbb9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:42.933277 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c335b5-fe29-4690-afc8-391c049bbb9d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:42.933277 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c335b5-fe29-4690-afc8-391c049bbb9d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:42.933446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c335b5-fe29-4690-afc8-391c049bbb9d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:42.947015 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c335b5-fe29-4690-afc8-391c049bbb9d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:42.948495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c335b5-fe29-4690-afc8-391c049bbb9d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:42.964438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-08c335b5-fe29-4690-afc8-391c049bbb9d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:42.964438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:42.964438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:42.964438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:42.964438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:42.964438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:42.964438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:42.964438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:42.964438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:42.964438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:42.964438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:42.964438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:42.964438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:42.964438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:42.964438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:42.964438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:42.965380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:42.965380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:42.965380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:42.965380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:42.965380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:42.965380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:42.965380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:42.965380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:42.965380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:42.965380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:42.965380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:42.965380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:42.965380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:42.965380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:42.965380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:42.965380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:42.965380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:42.966238 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:42.966238 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:42.966238 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:42.969638 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 239/472] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:42.971057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde165f1-7295-4364-94f4-fa0ef78954d0 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:42.971343 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde165f1-7295-4364-94f4-fa0ef78954d0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:42.971617 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde165f1-7295-4364-94f4-fa0ef78954d0 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:42.973139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 240/473] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:42.973838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde165f1-7295-4364-94f4-fa0ef78954d0 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:43.006620 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:43.010198 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2683dc68-93ac-455b-8bcf-feb9ef976a26 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:43.059871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2dd8e6-017c-4056-933d-ba9a52c0401e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:43.060479 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2dd8e6-017c-4056-933d-ba9a52c0401e None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:43.060751 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2dd8e6-017c-4056-933d-ba9a52c0401e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:43.061015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2dd8e6-017c-4056-933d-ba9a52c0401e None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:43.061653 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2dd8e6-017c-4056-933d-ba9a52c0401e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:43.063382 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2dd8e6-017c-4056-933d-ba9a52c0401e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:43.097476 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 234/474] 10.222.0.15 () {60 vars in 1132 bytes} [Tue Sep 13 01:05:42 2022] POST /identity/v3/users => generated 248 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:05:43.098779 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c771bf-d64e-4254-81c6-7736e7289d1c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:43.099105 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c771bf-d64e-4254-81c6-7736e7289d1c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:43.099427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c771bf-d64e-4254-81c6-7736e7289d1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:43.120233 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde165f1-7295-4364-94f4-fa0ef78954d0 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:43.120403 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-42c771bf-d64e-4254-81c6-7736e7289d1c None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:43.121549 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-42c771bf-d64e-4254-81c6-7736e7289d1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:43.199557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05586ab3-1190-4796-be79-1f60fc75f8bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:43.200190 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05586ab3-1190-4796-be79-1f60fc75f8bd None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:43.200461 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05586ab3-1190-4796-be79-1f60fc75f8bd None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:43.200729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05586ab3-1190-4796-be79-1f60fc75f8bd None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:43.201374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05586ab3-1190-4796-be79-1f60fc75f8bd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:43.201672 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05586ab3-1190-4796-be79-1f60fc75f8bd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:43.204791 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05586ab3-1190-4796-be79-1f60fc75f8bd None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:43.233624 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 241/475] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:43.235051 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:43.243523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05586ab3-1190-4796-be79-1f60fc75f8bd None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:43.264759 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 235/476] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:43.365574 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:43.383993 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05586ab3-1190-4796-be79-1f60fc75f8bd None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:43.427208 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b41e74-31fe-438a-93ed-34bbe6446dce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:43.427867 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b41e74-31fe-438a-93ed-34bbe6446dce None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:43.428143 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b41e74-31fe-438a-93ed-34bbe6446dce None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:43.428412 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b41e74-31fe-438a-93ed-34bbe6446dce None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:43.429047 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b41e74-31fe-438a-93ed-34bbe6446dce None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:43.430491 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b41e74-31fe-438a-93ed-34bbe6446dce None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:43.453786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 242/477] 10.222.0.15 () {60 vars in 1132 bytes} [Tue Sep 13 01:05:43 2022] POST /identity/v3/users => generated 249 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:05:43.557039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c771bf-d64e-4254-81c6-7736e7289d1c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:43.678281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c771bf-d64e-4254-81c6-7736e7289d1c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:43.713590 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cc6619-a09d-4a7d-b117-17b02dcc2eee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:43.714215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cc6619-a09d-4a7d-b117-17b02dcc2eee None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:43.714490 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cc6619-a09d-4a7d-b117-17b02dcc2eee None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:43.714762 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cc6619-a09d-4a7d-b117-17b02dcc2eee None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:43.715465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73cc6619-a09d-4a7d-b117-17b02dcc2eee None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:43.716846 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73cc6619-a09d-4a7d-b117-17b02dcc2eee None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:43.725822 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.sql.core [None req-73cc6619-a09d-4a7d-b117-17b02dcc2eee None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 13 01:05:43.725822 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 13 01:05:43.725822 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [parameters: {'id': '9625d0c48bdd4c0c8dbcfcd3b477d168', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 13 01:05:43.725822 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81404) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 13 01:05:43.743169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-73cc6619-a09d-4a7d-b117-17b02dcc2eee None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 13 01:05:43.743169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:43.743169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 13 01:05:43.743169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 13 01:05:43.743169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 13 01:05:43.743169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 13 01:05:43.743169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 13 01:05:43.743169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application result = self._query(query) Sep 13 01:05:43.743169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 13 01:05:43.743169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application conn.query(q) Sep 13 01:05:43.743169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 13 01:05:43.743169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 13 01:05:43.743169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 13 01:05:43.743169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application result.read() Sep 13 01:05:43.743169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 13 01:05:43.744065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 13 01:05:43.744065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 13 01:05:43.744065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 13 01:05:43.744065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 13 01:05:43.744065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 13 01:05:43.744065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 13 01:05:43.744065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 13 01:05:43.744065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 13 01:05:43.744065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:43.744065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 13 01:05:43.744065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:43.744065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:43.744065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 13 01:05:43.744065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 13 01:05:43.744065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 13 01:05:43.744065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return ref.to_dict() Sep 13 01:05:43.744065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 13 01:05:43.744065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application next(self.gen) Sep 13 01:05:43.745053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 13 01:05:43.745053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application yield resource Sep 13 01:05:43.745053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 13 01:05:43.745053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application next(self.gen) Sep 13 01:05:43.745053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 13 01:05:43.745053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self.session.rollback() Sep 13 01:05:43.745053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 13 01:05:43.745053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self.force_reraise() Sep 13 01:05:43.745053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 13 01:05:43.745053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise self.value Sep 13 01:05:43.745053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 13 01:05:43.745053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 13 01:05:43.745053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 13 01:05:43.745053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application session.commit() Sep 13 01:05:43.745053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 13 01:05:43.745053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 13 01:05:43.745053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 13 01:05:43.745907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._prepare_impl() Sep 13 01:05:43.745907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 13 01:05:43.745907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self.session.flush() Sep 13 01:05:43.745907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 13 01:05:43.745907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._flush(objects) Sep 13 01:05:43.745907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 13 01:05:43.745907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 13 01:05:43.745907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 13 01:05:43.745907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application compat.raise_( Sep 13 01:05:43.745907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 13 01:05:43.745907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception Sep 13 01:05:43.745907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 13 01:05:43.745907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application flush_context.execute() Sep 13 01:05:43.745907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 13 01:05:43.745907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rec.execute(self) Sep 13 01:05:43.745907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 13 01:05:43.745907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 13 01:05:43.746784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 13 01:05:43.746784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 13 01:05:43.746784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 13 01:05:43.746784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 13 01:05:43.746784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 13 01:05:43.746784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 13 01:05:43.746784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 13 01:05:43.746784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 13 01:05:43.746784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 13 01:05:43.746784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 13 01:05:43.746784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 13 01:05:43.746784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 13 01:05:43.746784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 13 01:05:43.746784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 13 01:05:43.746784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 13 01:05:43.746784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception Sep 13 01:05:43.747635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 13 01:05:43.747635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 13 01:05:43.747635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 13 01:05:43.747635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 13 01:05:43.747635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 13 01:05:43.747635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application result = self._query(query) Sep 13 01:05:43.747635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 13 01:05:43.747635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application conn.query(q) Sep 13 01:05:43.747635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 13 01:05:43.747635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 13 01:05:43.747635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 13 01:05:43.747635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application result.read() Sep 13 01:05:43.747635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 13 01:05:43.747635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 13 01:05:43.747635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 13 01:05:43.747635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 13 01:05:43.748406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 13 01:05:43.748406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 13 01:05:43.748406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 13 01:05:43.748406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 13 01:05:43.748406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 13 01:05:43.748406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 13 01:05:43.748406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [parameters: {'id': '9625d0c48bdd4c0c8dbcfcd3b477d168', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 13 01:05:43.748406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 13 01:05:43.748406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:43.748406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:05:43.748406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:43.748406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:43.748406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:43.748406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:43.748406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:43.748406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:43.749187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:43.749187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:43.749187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:43.749187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:43.749187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:43.749187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:43.749187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 13 01:05:43.749187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 13 01:05:43.749187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:43.749187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:43.749187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 13 01:05:43.749187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 13 01:05:43.749187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 13 01:05:43.749187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 13 01:05:43.749187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 13 01:05:43.749187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:43.750030 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 236/478] 10.222.0.15 () {60 vars in 1131 bytes} [Tue Sep 13 01:05:43 2022] POST /identity/v3/roles => generated 144 bytes in 188 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:43.755979 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b41e74-31fe-438a-93ed-34bbe6446dce None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:43.877840 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b41e74-31fe-438a-93ed-34bbe6446dce None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:43.916222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c20e3a-7cda-462f-b146-163ebde867b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:43.916860 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c20e3a-7cda-462f-b146-163ebde867b1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:43.917133 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c20e3a-7cda-462f-b146-163ebde867b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:43.917403 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c20e3a-7cda-462f-b146-163ebde867b1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:43.925375 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:43.930068 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c20e3a-7cda-462f-b146-163ebde867b1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:43.934903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-b3c20e3a-7cda-462f-b146-163ebde867b1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:43.934903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:43.934903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:43.934903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:43.934903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:43.934903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:43.934903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:43.934903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:43.934903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:43.934903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:43.934903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:43.934903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:43.934903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:43.934903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:43.934903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:43.934903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:43.936975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:43.936975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:43.936975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:43.936975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:43.936975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:43.936975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:43.936975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:43.936975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:43.936975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:43.936975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:43.936975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:43.936975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:43.936975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:43.936975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:43.936975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:43.936975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:43.940861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:43.940861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:43.940861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:43.940861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:43.940861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:43.940861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:43.940861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:43.940861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:43.940861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:43.940861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:43.940861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 01:05:43.940861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:43.940861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 243/479] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:05:43 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:43.942545 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cc6619-a09d-4a7d-b117-17b02dcc2eee None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:44.018398 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:44.087281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cc6619-a09d-4a7d-b117-17b02dcc2eee None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:44.134119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b95be9-9aba-4865-a7d9-0444d0bdbf7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:44.134734 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b95be9-9aba-4865-a7d9-0444d0bdbf7a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:44.135037 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b95be9-9aba-4865-a7d9-0444d0bdbf7a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:44.135307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b95be9-9aba-4865-a7d9-0444d0bdbf7a None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:44.135969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b95be9-9aba-4865-a7d9-0444d0bdbf7a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:44.136267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b95be9-9aba-4865-a7d9-0444d0bdbf7a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:44.137640 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b95be9-9aba-4865-a7d9-0444d0bdbf7a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:44.151057 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 237/480] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:43 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:45.010791 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b961ac-6239-4bb7-b98f-c291a32ca17c None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:45.011846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b961ac-6239-4bb7-b98f-c291a32ca17c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:45.012328 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b961ac-6239-4bb7-b98f-c291a32ca17c None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:45.014119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 244/481] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:45.019875 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f93aed9-3878-4699-a094-d3103e642061 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:45.020275 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f93aed9-3878-4699-a094-d3103e642061 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:45.020446 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f93aed9-3878-4699-a094-d3103e642061 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:45.045892 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-0f93aed9-3878-4699-a094-d3103e642061 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:45.047125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0f93aed9-3878-4699-a094-d3103e642061 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:45.139259 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 238/482] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:45.386017 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eff011f-4777-4e42-9e1d-2633f291effd None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:45.386347 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eff011f-4777-4e42-9e1d-2633f291effd None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:45.386517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eff011f-4777-4e42-9e1d-2633f291effd None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:45.388062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 245/483] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:45.393222 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47577ea-45c9-4c33-a605-d5374ad5a8c5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:45.393510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47577ea-45c9-4c33-a605-d5374ad5a8c5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:45.393784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47577ea-45c9-4c33-a605-d5374ad5a8c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:45.417933 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d47577ea-45c9-4c33-a605-d5374ad5a8c5 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:45.419228 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d47577ea-45c9-4c33-a605-d5374ad5a8c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:45.468205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eff011f-4777-4e42-9e1d-2633f291effd None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:45.484334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:45.526079 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 239/484] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:45.624199 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eff011f-4777-4e42-9e1d-2633f291effd None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:45.676557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f91013-8c74-414e-9077-e06ea45d4e3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:45.677098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f91013-8c74-414e-9077-e06ea45d4e3f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:45.677373 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f91013-8c74-414e-9077-e06ea45d4e3f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:45.677638 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f91013-8c74-414e-9077-e06ea45d4e3f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:45.695756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f91013-8c74-414e-9077-e06ea45d4e3f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:45.700894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-04f91013-8c74-414e-9077-e06ea45d4e3f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:45.700894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:45.700894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:45.700894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:45.700894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:45.700894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:45.700894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:45.700894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:45.700894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:45.700894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:45.700894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:45.700894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:45.700894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:45.700894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:45.700894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:45.700894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:45.701919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:45.701919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:45.701919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:45.701919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:45.701919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:45.701919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:45.701919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:45.701919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:45.701919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:45.701919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:45.701919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:45.701919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:45.701919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:45.701919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:45.701919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:45.701919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:45.703065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:45.703065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:45.703065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:45.703065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:45.703065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:45.703065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:45.703065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:45.703065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:45.703065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:45.703065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:45.703065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:45.703065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:45.703065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 246/485] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:05:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:45.708964 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47577ea-45c9-4c33-a605-d5374ad5a8c5 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:45.857963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f91013-8c74-414e-9077-e06ea45d4e3f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:45.875557 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47577ea-45c9-4c33-a605-d5374ad5a8c5 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:45.912474 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1a2ced-405e-4466-bb65-96316d30db7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:45.913376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1a2ced-405e-4466-bb65-96316d30db7f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:45.914395 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1a2ced-405e-4466-bb65-96316d30db7f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:45.915141 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1a2ced-405e-4466-bb65-96316d30db7f None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:45.916024 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1a2ced-405e-4466-bb65-96316d30db7f None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:45.916536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1a2ced-405e-4466-bb65-96316d30db7f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:45.918191 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1a2ced-405e-4466-bb65-96316d30db7f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:45.927998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 240/486] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:05:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:45.935884 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1a2ced-405e-4466-bb65-96316d30db7f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:46.005782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:46.008982 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:46.062275 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f91013-8c74-414e-9077-e06ea45d4e3f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:46.071421 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1a2ced-405e-4466-bb65-96316d30db7f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:46.105485 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd98851-ac8b-4444-b758-73de9927c1aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:46.106381 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd98851-ac8b-4444-b758-73de9927c1aa None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:46.106814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd98851-ac8b-4444-b758-73de9927c1aa None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:46.107303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd98851-ac8b-4444-b758-73de9927c1aa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:46.109273 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621de709-9e44-466a-abab-e4e757d3c783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:46.110229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621de709-9e44-466a-abab-e4e757d3c783 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:46.110989 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621de709-9e44-466a-abab-e4e757d3c783 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:46.111430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621de709-9e44-466a-abab-e4e757d3c783 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:46.119805 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd98851-ac8b-4444-b758-73de9927c1aa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:46.121376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd98851-ac8b-4444-b758-73de9927c1aa None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:46.124776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621de709-9e44-466a-abab-e4e757d3c783 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:46.131377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-621de709-9e44-466a-abab-e4e757d3c783 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:46.131377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:46.131377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:46.131377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:46.131377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:46.131377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:46.131377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:46.131377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:46.131377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:46.131377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:46.131377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:46.131377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:46.131377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:46.131377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:46.131377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:46.131377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:46.132997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:46.132997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:46.132997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:46.132997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:46.132997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:46.132997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:46.132997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:46.132997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:46.132997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:46.132997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:46.132997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:46.132997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:46.132997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:46.132997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:46.132997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:46.132997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:46.133924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:46.133924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:46.133924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:46.133924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:46.133924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:46.133924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:46.133924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:46.133924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:46.133924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:46.133924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:46.133924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:46.133924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:46.136242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 247/487] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:05:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:05:46.137578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944eaba7-8789-4d9b-a701-e9426d2b230c None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:46.138022 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944eaba7-8789-4d9b-a701-e9426d2b230c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:46.138471 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944eaba7-8789-4d9b-a701-e9426d2b230c None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:46.139951 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 248/488] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:46.144040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-5dd98851-ac8b-4444-b758-73de9927c1aa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:46.144040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:46.144040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:46.144040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:46.144040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:46.144040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:46.144040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:46.144040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:46.144040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:46.144040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:46.144040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:46.144040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:46.144040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:46.144040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:46.144040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:46.144040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:46.144894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:46.144894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:46.144894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:46.144894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:46.144894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:46.144894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:46.144894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:46.144894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:46.144894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:46.144894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:46.144894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:46.144894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:46.144894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:46.144894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:46.144894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:46.144894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:46.144894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:46.145738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:46.145738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:46.145738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:46.147145 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944eaba7-8789-4d9b-a701-e9426d2b230c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:46.149368 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 241/489] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:46.150646 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfa3351-8a6f-42d9-ade9-b9fe52289c26 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:46.150996 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfa3351-8a6f-42d9-ade9-b9fe52289c26 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:46.151271 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfa3351-8a6f-42d9-ade9-b9fe52289c26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:46.169658 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-dcfa3351-8a6f-42d9-ade9-b9fe52289c26 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:46.170936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-dcfa3351-8a6f-42d9-ade9-b9fe52289c26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:46.264630 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 242/490] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:46.264630 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfa3351-8a6f-42d9-ade9-b9fe52289c26 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:46.283161 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944eaba7-8789-4d9b-a701-e9426d2b230c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:46.309893 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:46.315379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:46.345764 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed93bb36-922a-45f4-a11c-43d3fdce40b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:46.346376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed93bb36-922a-45f4-a11c-43d3fdce40b5 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:46.346651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed93bb36-922a-45f4-a11c-43d3fdce40b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:46.346953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed93bb36-922a-45f4-a11c-43d3fdce40b5 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:46.347611 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed93bb36-922a-45f4-a11c-43d3fdce40b5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:46.347913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed93bb36-922a-45f4-a11c-43d3fdce40b5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:46.350928 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed93bb36-922a-45f4-a11c-43d3fdce40b5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:46.368799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 249/491] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:05:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:46.376419 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed93bb36-922a-45f4-a11c-43d3fdce40b5 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:46.465716 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfa3351-8a6f-42d9-ade9-b9fe52289c26 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:46.515498 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c2640c-d51e-4b6d-8ad2-609849d7112c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:46.516253 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c2640c-d51e-4b6d-8ad2-609849d7112c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:46.516783 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c2640c-d51e-4b6d-8ad2-609849d7112c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:46.517194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c2640c-d51e-4b6d-8ad2-609849d7112c None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:46.517998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c2640c-d51e-4b6d-8ad2-609849d7112c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:46.518442 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c2640c-d51e-4b6d-8ad2-609849d7112c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:46.520312 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c2640c-d51e-4b6d-8ad2-609849d7112c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:46.534202 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 243/492] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:46.536365 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43d4408-bc1e-4f93-8585-da925947792b None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:46.536657 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43d4408-bc1e-4f93-8585-da925947792b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:46.536929 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43d4408-bc1e-4f93-8585-da925947792b None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:46.542292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 244/493] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:46.542292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43d4408-bc1e-4f93-8585-da925947792b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:46.546229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed93bb36-922a-45f4-a11c-43d3fdce40b5 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:46.592405 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a292a0e5-cb84-446f-a639-e6f380cbdefa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:46.593028 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a292a0e5-cb84-446f-a639-e6f380cbdefa None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:46.593300 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a292a0e5-cb84-446f-a639-e6f380cbdefa None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:46.593565 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a292a0e5-cb84-446f-a639-e6f380cbdefa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:46.609444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a292a0e5-cb84-446f-a639-e6f380cbdefa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:46.611871 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a292a0e5-cb84-446f-a639-e6f380cbdefa None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:46.627383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-a292a0e5-cb84-446f-a639-e6f380cbdefa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:46.627383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:46.627383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:46.627383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:46.627383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:46.627383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:46.627383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:46.627383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:46.627383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:46.627383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:46.627383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:46.627383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:46.627383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:46.627383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:46.627383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:46.627383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:46.628304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:46.628304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:46.628304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:46.628304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:46.628304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:46.628304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:46.628304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:46.628304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:46.628304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:46.628304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:46.628304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:46.628304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:46.628304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:46.628304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:46.628304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:46.628304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:46.628304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:46.629169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:46.629169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:46.629169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:46.629169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 250/494] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:46.629915 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e591a1df-53a0-422c-a6e0-0678d6945578 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:46.630181 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e591a1df-53a0-422c-a6e0-0678d6945578 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:46.630449 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e591a1df-53a0-422c-a6e0-0678d6945578 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:46.647825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:46.650069 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e591a1df-53a0-422c-a6e0-0678d6945578 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:46.651518 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e591a1df-53a0-422c-a6e0-0678d6945578 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:46.669396 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43d4408-bc1e-4f93-8585-da925947792b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:46.678795 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:46.705146 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b2b753-88da-4353-ad87-abe05e5c45f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:46.705775 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b2b753-88da-4353-ad87-abe05e5c45f4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:46.706047 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b2b753-88da-4353-ad87-abe05e5c45f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:46.706370 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b2b753-88da-4353-ad87-abe05e5c45f4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:46.713861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b2b753-88da-4353-ad87-abe05e5c45f4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:46.715330 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b2b753-88da-4353-ad87-abe05e5c45f4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:46.722563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 251/495] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:46.723668 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e591a1df-53a0-422c-a6e0-0678d6945578 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:46.728219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-d7b2b753-88da-4353-ad87-abe05e5c45f4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 01:05:46.728219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:46.728219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:46.728219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:46.728219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:46.728219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:46.728219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:46.728219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:46.728219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:46.728219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:46.728219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:46.728219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:46.728219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:05:46.728219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:05:46.728219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:05:46.728219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:05:46.729104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:46.729104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:46.729104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:05:46.729104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:46.729104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:05:46.729104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:46.729104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:46.729104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:46.729104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:46.729104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:46.729104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:46.729104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:46.729104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:46.729104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:46.729104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:46.729104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:46.729104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:46.729104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:46.730068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:46.730068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:46.730068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:05:46.730068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:05:46.730068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:05:46.730068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:05:46.730068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:05:46.730068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:05:46.730068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 01:05:46.730068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:46.733904 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 245/496] 10.222.0.15 () {56 vars in 1093 bytes} [Tue Sep 13 01:05:46 2022] GET /identity/v3/users/nova => generated 82 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:46.734744 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b2b753-88da-4353-ad87-abe05e5c45f4 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:46.843769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b2b753-88da-4353-ad87-abe05e5c45f4 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:46.864068 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e591a1df-53a0-422c-a6e0-0678d6945578 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:46.879744 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8dc053-f11c-47ae-857c-b6d02af93994 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:46.880471 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8dc053-f11c-47ae-857c-b6d02af93994 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:46.880818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8dc053-f11c-47ae-857c-b6d02af93994 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:46.881154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8dc053-f11c-47ae-857c-b6d02af93994 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:46.881880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8dc053-f11c-47ae-857c-b6d02af93994 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:46.882255 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8dc053-f11c-47ae-857c-b6d02af93994 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:46.884311 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8dc053-f11c-47ae-857c-b6d02af93994 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:46.898257 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 246/497] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:46.899385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8dc053-f11c-47ae-857c-b6d02af93994 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:46.902412 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:46.914217 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6657be8a-3767-41cf-ad17-4eb77f42dfb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:46.915660 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6657be8a-3767-41cf-ad17-4eb77f42dfb0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:46.916405 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6657be8a-3767-41cf-ad17-4eb77f42dfb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:46.916918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6657be8a-3767-41cf-ad17-4eb77f42dfb0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:46.932840 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6657be8a-3767-41cf-ad17-4eb77f42dfb0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:46.940444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-6657be8a-3767-41cf-ad17-4eb77f42dfb0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:46.940444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:46.940444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:46.940444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:46.940444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:46.940444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:46.940444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:46.940444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:46.940444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:46.940444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:46.940444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:46.940444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:46.940444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:46.940444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:46.940444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:46.940444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:46.941298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:46.941298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:46.941298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:46.941298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:46.941298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:46.941298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:46.941298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:46.941298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:46.941298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:46.941298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:46.941298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:46.941298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:46.941298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:46.941298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:46.941298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:46.941298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:46.942085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:46.942085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:46.942085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:46.942085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:46.942085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:46.942085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:46.942085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:46.942085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:46.942085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:46.942085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:46.942085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:46.942085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:46.944602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 252/498] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:05:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:46.946235 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6657be8a-3767-41cf-ad17-4eb77f42dfb0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:46.981291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:47.010272 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8dc053-f11c-47ae-857c-b6d02af93994 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:47.048241 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9f833f-6499-4409-85c9-cd268d422ea0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:47.048845 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9f833f-6499-4409-85c9-cd268d422ea0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:47.049118 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9f833f-6499-4409-85c9-cd268d422ea0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:47.049388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9f833f-6499-4409-85c9-cd268d422ea0 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:47.050100 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9f833f-6499-4409-85c9-cd268d422ea0 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:47.050407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9f833f-6499-4409-85c9-cd268d422ea0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:47.051857 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9f833f-6499-4409-85c9-cd268d422ea0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:47.066732 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6657be8a-3767-41cf-ad17-4eb77f42dfb0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:47.088095 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 247/499] 10.222.0.15 () {56 vars in 1133 bytes} [Tue Sep 13 01:05:46 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:47.088927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9f833f-6499-4409-85c9-cd268d422ea0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:47.102817 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43aa1b10-2591-416b-a4c8-83d1c67fd0a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:47.103430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aa1b10-2591-416b-a4c8-83d1c67fd0a0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:47.103705 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aa1b10-2591-416b-a4c8-83d1c67fd0a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:47.103972 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aa1b10-2591-416b-a4c8-83d1c67fd0a0 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:47.110940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43aa1b10-2591-416b-a4c8-83d1c67fd0a0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:47.112370 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43aa1b10-2591-416b-a4c8-83d1c67fd0a0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:47.126088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-43aa1b10-2591-416b-a4c8-83d1c67fd0a0 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 13 01:05:47.126088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:47.126088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:47.126088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:47.126088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:47.126088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:47.126088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:47.126088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:47.126088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:47.126088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:47.126088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:47.126088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:47.126088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:05:47.126088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:05:47.126088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:05:47.126088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:05:47.127179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:47.127179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:47.127179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:05:47.127179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:47.127179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:05:47.127179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:47.127179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:47.127179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:47.127179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:47.127179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:47.127179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:47.127179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:47.127179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:47.127179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:47.127179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:47.127179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:47.127179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:47.127179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:47.128055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:47.128055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:47.128055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:05:47.128055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:05:47.128055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:05:47.128055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:05:47.128055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:05:47.128055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:05:47.128055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 13 01:05:47.128055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:47.128055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 253/500] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:46 2022] GET /identity/v3/users/glance => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:47.128797 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279c2769-e9e8-4d2c-9b02-e6fe4a951bd8 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:47.128958 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279c2769-e9e8-4d2c-9b02-e6fe4a951bd8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:47.129231 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279c2769-e9e8-4d2c-9b02-e6fe4a951bd8 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:47.130424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 254/501] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:47 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:47.131279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279c2769-e9e8-4d2c-9b02-e6fe4a951bd8 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:47.151207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:47.191625 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9f833f-6499-4409-85c9-cd268d422ea0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:47.214372 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:47.225418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c76d189-b2aa-42a2-86d9-d001fb1499cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:47.226028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c76d189-b2aa-42a2-86d9-d001fb1499cb None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:47.226377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c76d189-b2aa-42a2-86d9-d001fb1499cb None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:47.226647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c76d189-b2aa-42a2-86d9-d001fb1499cb None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:47.227325 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c76d189-b2aa-42a2-86d9-d001fb1499cb None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:47.227621 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c76d189-b2aa-42a2-86d9-d001fb1499cb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:47.228973 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c76d189-b2aa-42a2-86d9-d001fb1499cb None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:47.231211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279c2769-e9e8-4d2c-9b02-e6fe4a951bd8 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:47.236635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 248/502] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:05:47 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:47.237404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c76d189-b2aa-42a2-86d9-d001fb1499cb None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:47.265025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21faba04-1270-47cf-8b3e-c7265ee226fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:47.265617 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21faba04-1270-47cf-8b3e-c7265ee226fe None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:47.265870 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21faba04-1270-47cf-8b3e-c7265ee226fe None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:47.266139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21faba04-1270-47cf-8b3e-c7265ee226fe None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:47.277065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21faba04-1270-47cf-8b3e-c7265ee226fe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:47.281842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-21faba04-1270-47cf-8b3e-c7265ee226fe None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:47.281842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:47.281842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:47.281842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:47.281842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:47.281842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:47.281842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:47.281842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:47.281842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:47.281842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:47.281842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:47.281842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:47.281842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:47.281842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:47.281842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:47.281842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:47.282919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:47.282919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:47.282919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:47.282919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:47.282919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:47.282919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:47.282919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:47.282919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:47.282919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:47.282919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:47.282919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:47.282919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:47.282919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:47.282919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:47.282919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:47.282919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:47.283722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:47.283722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:47.283722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:47.283722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:47.283722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:47.283722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:47.283722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:47.283722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:47.283722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:47.283722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:47.283722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:47.283722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:47.283722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 255/503] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:05:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:47.283722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21faba04-1270-47cf-8b3e-c7265ee226fe None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:47.340176 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c76d189-b2aa-42a2-86d9-d001fb1499cb None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:47.361925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:47.373756 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ef872e-654f-45fa-8551-69352f1edeb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:47.374360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ef872e-654f-45fa-8551-69352f1edeb9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:47.374637 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ef872e-654f-45fa-8551-69352f1edeb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:47.374936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ef872e-654f-45fa-8551-69352f1edeb9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:47.387812 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21faba04-1270-47cf-8b3e-c7265ee226fe None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:47.421540 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78213e82-d64b-49a7-b49d-65eb20991e7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:47.422138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78213e82-d64b-49a7-b49d-65eb20991e7a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:47.422414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78213e82-d64b-49a7-b49d-65eb20991e7a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:47.422681 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78213e82-d64b-49a7-b49d-65eb20991e7a None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:47.423425 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78213e82-d64b-49a7-b49d-65eb20991e7a None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:47.423727 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78213e82-d64b-49a7-b49d-65eb20991e7a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:47.425157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78213e82-d64b-49a7-b49d-65eb20991e7a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:47.443294 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ef872e-654f-45fa-8551-69352f1edeb9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:47.444742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ef872e-654f-45fa-8551-69352f1edeb9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:47.461894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 256/504] 10.222.0.15 () {56 vars in 1137 bytes} [Tue Sep 13 01:05:47 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:47.463119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6be5bed-acc3-4348-9c45-4ab3706b6056 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:47.463404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6be5bed-acc3-4348-9c45-4ab3706b6056 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:47.463676 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6be5bed-acc3-4348-9c45-4ab3706b6056 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:47.469708 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:47.482752 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a6be5bed-acc3-4348-9c45-4ab3706b6056 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:47.483971 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a6be5bed-acc3-4348-9c45-4ab3706b6056 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:47.541116 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 249/505] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:47.541939 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ef872e-654f-45fa-8551-69352f1edeb9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:47.553041 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 257/506] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:47.554009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6be5bed-acc3-4348-9c45-4ab3706b6056 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:47.574627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:47.652103 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ef872e-654f-45fa-8551-69352f1edeb9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:47.665195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6be5bed-acc3-4348-9c45-4ab3706b6056 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:47.688662 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bdf175-89bd-4fc1-a3aa-813a2c6eb375 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:47.689229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bdf175-89bd-4fc1-a3aa-813a2c6eb375 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:47.689507 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bdf175-89bd-4fc1-a3aa-813a2c6eb375 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:47.689771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bdf175-89bd-4fc1-a3aa-813a2c6eb375 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:47.696626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:47.700752 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19bdf175-89bd-4fc1-a3aa-813a2c6eb375 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:47.702175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fb1548-4718-4dad-996e-a39ced460b0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:47.702313 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19bdf175-89bd-4fc1-a3aa-813a2c6eb375 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:47.702771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fb1548-4718-4dad-996e-a39ced460b0c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:47.703071 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fb1548-4718-4dad-996e-a39ced460b0c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:47.703341 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fb1548-4718-4dad-996e-a39ced460b0c None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:47.703990 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fb1548-4718-4dad-996e-a39ced460b0c None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:47.704288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fb1548-4718-4dad-996e-a39ced460b0c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:47.705623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fb1548-4718-4dad-996e-a39ced460b0c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:47.719022 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 258/507] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:05:47 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:47.719022 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fb1548-4718-4dad-996e-a39ced460b0c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:47.719224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-19bdf175-89bd-4fc1-a3aa-813a2c6eb375 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:47.719224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:47.719224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:47.719224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:47.719224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:47.719224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:47.719224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:47.719224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:47.719224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:47.719224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:47.719224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:47.719224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:47.719224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:47.719224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:47.719224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:47.719224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:47.720039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:47.720039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:47.720039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:47.720039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:47.720039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:47.720039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:47.720039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:47.720039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:47.720039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:47.720039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:47.720039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:47.720039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:47.720039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:47.720039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:47.720039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:47.720039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:47.720039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:47.720869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:47.720869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:47.720869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:47.720869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 250/508] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:47.720869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bdf175-89bd-4fc1-a3aa-813a2c6eb375 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:47.796305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:47.823298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bdf175-89bd-4fc1-a3aa-813a2c6eb375 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:47.827250 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fb1548-4718-4dad-996e-a39ced460b0c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:47.857186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2720386f-76a0-4d36-9644-ff13596c4557 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:47.857814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2720386f-76a0-4d36-9644-ff13596c4557 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:47.858087 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2720386f-76a0-4d36-9644-ff13596c4557 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:47.858353 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2720386f-76a0-4d36-9644-ff13596c4557 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:47.861119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474c17ba-546c-42da-bdf8-7fcaa34b1ea5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:47.861707 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474c17ba-546c-42da-bdf8-7fcaa34b1ea5 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:47.861979 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474c17ba-546c-42da-bdf8-7fcaa34b1ea5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:47.862246 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474c17ba-546c-42da-bdf8-7fcaa34b1ea5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:47.868536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2720386f-76a0-4d36-9644-ff13596c4557 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:47.869993 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2720386f-76a0-4d36-9644-ff13596c4557 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:47.883836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-2720386f-76a0-4d36-9644-ff13596c4557 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:47.883836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:47.883836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:47.883836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:47.883836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:47.883836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:47.883836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:47.883836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:47.883836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:47.883836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:47.883836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:47.883836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:47.883836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:47.883836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:47.883836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:47.883836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:47.884720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:47.884720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:47.884720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:47.884720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:47.884720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:47.884720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:47.884720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:47.884720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:47.884720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:47.884720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:47.884720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:47.884720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:47.884720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:47.884720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:47.884720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:47.884720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:47.884720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:47.885588 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:47.885588 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:47.885588 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:47.885588 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 251/509] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:47.885823 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2720386f-76a0-4d36-9644-ff13596c4557 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:47.926813 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:47.934358 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474c17ba-546c-42da-bdf8-7fcaa34b1ea5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:47.935834 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474c17ba-546c-42da-bdf8-7fcaa34b1ea5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:47.984954 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2720386f-76a0-4d36-9644-ff13596c4557 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:48.008882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:48.018435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1349d6f9-257c-4a80-92b2-1d475757bf6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:48.019083 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1349d6f9-257c-4a80-92b2-1d475757bf6d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:48.019363 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1349d6f9-257c-4a80-92b2-1d475757bf6d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:48.019633 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1349d6f9-257c-4a80-92b2-1d475757bf6d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:48.030688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1349d6f9-257c-4a80-92b2-1d475757bf6d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:48.032141 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1349d6f9-257c-4a80-92b2-1d475757bf6d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:48.034680 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 259/510] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:48.035536 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474c17ba-546c-42da-bdf8-7fcaa34b1ea5 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:48.047699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-1349d6f9-257c-4a80-92b2-1d475757bf6d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:48.047699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:48.047699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:48.047699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:48.047699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:48.047699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:48.047699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:48.047699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:48.047699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:48.047699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:48.047699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:48.047699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:48.047699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:48.047699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:48.047699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:48.047699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:48.048623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:48.048623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:48.048623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:48.048623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:48.048623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:48.048623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:48.048623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:48.048623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:48.048623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:48.048623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:48.048623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:48.048623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:48.048623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:48.048623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:48.048623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:48.048623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:48.048623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:48.049443 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:48.049443 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:48.049443 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:48.049443 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 252/511] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:48.049661 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1349d6f9-257c-4a80-92b2-1d475757bf6d None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:48.129033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:48.136623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474c17ba-546c-42da-bdf8-7fcaa34b1ea5 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:48.148375 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1349d6f9-257c-4a80-92b2-1d475757bf6d None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:48.169636 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41a2e6f-6109-429a-b0d3-069328b3e85d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:48.170234 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41a2e6f-6109-429a-b0d3-069328b3e85d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:48.170504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41a2e6f-6109-429a-b0d3-069328b3e85d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:48.170778 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41a2e6f-6109-429a-b0d3-069328b3e85d None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:48.171420 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41a2e6f-6109-429a-b0d3-069328b3e85d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:48.171719 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41a2e6f-6109-429a-b0d3-069328b3e85d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:48.173056 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41a2e6f-6109-429a-b0d3-069328b3e85d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:48.180862 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e675c12-bd28-4c37-bbe9-5319c7f981e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:48.181505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e675c12-bd28-4c37-bbe9-5319c7f981e9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:48.181786 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e675c12-bd28-4c37-bbe9-5319c7f981e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:48.182058 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e675c12-bd28-4c37-bbe9-5319c7f981e9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:48.184188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 260/512] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:48.184965 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41a2e6f-6109-429a-b0d3-069328b3e85d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:48.192981 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e675c12-bd28-4c37-bbe9-5319c7f981e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:48.194417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e675c12-bd28-4c37-bbe9-5319c7f981e9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:48.208671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-3e675c12-bd28-4c37-bbe9-5319c7f981e9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:48.208671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:48.208671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:48.208671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:48.208671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:48.208671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:48.208671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:48.208671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:48.208671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:48.208671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:48.208671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:48.208671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:48.208671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:48.208671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:48.208671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:48.208671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:48.209736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:48.209736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:48.209736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:48.209736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:48.209736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:48.209736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:48.209736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:48.209736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:48.209736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:48.209736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:48.209736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:48.209736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:48.209736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:48.209736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:48.209736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:48.209736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:48.209736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:48.210626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:48.210626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:48.210626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:48.210626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 253/513] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:48.210626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e675c12-bd28-4c37-bbe9-5319c7f981e9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:48.240101 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:48.285727 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41a2e6f-6109-429a-b0d3-069328b3e85d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:48.306331 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e675c12-bd28-4c37-bbe9-5319c7f981e9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:48.318301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f02830-3d77-4e91-b545-1c04eec0e3fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:48.318921 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f02830-3d77-4e91-b545-1c04eec0e3fa None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:48.319201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f02830-3d77-4e91-b545-1c04eec0e3fa None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:48.319469 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f02830-3d77-4e91-b545-1c04eec0e3fa None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:48.322436 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f02830-3d77-4e91-b545-1c04eec0e3fa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:48.322739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f02830-3d77-4e91-b545-1c04eec0e3fa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:48.324136 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f02830-3d77-4e91-b545-1c04eec0e3fa None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:48.336302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 261/514] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:48.337281 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f02830-3d77-4e91-b545-1c04eec0e3fa None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:48.339612 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078a5412-63d6-4d72-86e9-b4697db9c4c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:48.340256 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078a5412-63d6-4d72-86e9-b4697db9c4c3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:48.340711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078a5412-63d6-4d72-86e9-b4697db9c4c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:48.340819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078a5412-63d6-4d72-86e9-b4697db9c4c3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:48.351566 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078a5412-63d6-4d72-86e9-b4697db9c4c3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:48.353000 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078a5412-63d6-4d72-86e9-b4697db9c4c3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:48.353481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:48.367361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-078a5412-63d6-4d72-86e9-b4697db9c4c3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:48.367361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:48.367361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:48.367361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:48.367361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:48.367361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:48.367361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:48.367361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:48.367361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:48.367361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:48.367361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:48.367361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:48.367361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:48.367361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:48.367361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:48.367361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:48.372776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:48.372776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:48.372776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:48.372776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:48.372776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:48.372776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:48.372776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:48.372776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:48.372776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:48.372776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:48.372776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:48.372776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:48.372776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:48.372776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:48.372776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:48.372776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:48.372776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:48.373880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:48.373880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:48.373880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:48.373880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 254/515] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:48.373880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078a5412-63d6-4d72-86e9-b4697db9c4c3 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:48.440401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f02830-3d77-4e91-b545-1c04eec0e3fa None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:48.450511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:48.468836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078a5412-63d6-4d72-86e9-b4697db9c4c3 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:48.474631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58cbf57-90d7-4e17-8607-00c3797a4ecc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:48.475275 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58cbf57-90d7-4e17-8607-00c3797a4ecc None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:48.475554 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58cbf57-90d7-4e17-8607-00c3797a4ecc None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:48.475823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58cbf57-90d7-4e17-8607-00c3797a4ecc None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:48.476501 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58cbf57-90d7-4e17-8607-00c3797a4ecc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:48.476800 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58cbf57-90d7-4e17-8607-00c3797a4ecc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:48.478140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58cbf57-90d7-4e17-8607-00c3797a4ecc None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:48.493763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 262/516] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:48.493763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58cbf57-90d7-4e17-8607-00c3797a4ecc None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:48.501593 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab902b36-eaac-4769-b6df-e8ea4d67ffad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:48.502214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab902b36-eaac-4769-b6df-e8ea4d67ffad None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:48.502486 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab902b36-eaac-4769-b6df-e8ea4d67ffad None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:48.502755 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab902b36-eaac-4769-b6df-e8ea4d67ffad None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:48.510264 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab902b36-eaac-4769-b6df-e8ea4d67ffad None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:48.511761 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab902b36-eaac-4769-b6df-e8ea4d67ffad None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:48.522613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-ab902b36-eaac-4769-b6df-e8ea4d67ffad None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 01:05:48.522613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:48.522613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:48.522613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:48.522613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:48.522613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:48.522613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:48.522613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:48.522613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:48.522613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:48.522613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:48.522613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:48.522613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:05:48.522613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:05:48.522613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:05:48.522613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:05:48.524213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:48.524213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:48.524213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:05:48.524213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:48.524213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:05:48.524213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:48.524213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:48.524213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:48.524213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:48.524213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:48.524213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:48.524213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:48.524213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:48.524213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:48.524213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:48.524213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:48.524213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:48.524213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:48.525082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:48.525082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:48.525082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:05:48.525082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:05:48.525082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:05:48.525082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:05:48.525082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:05:48.525082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:05:48.525082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 01:05:48.525082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:48.525082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 255/517] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:48 2022] GET /identity/v3/users/cinder => generated 84 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:48.525082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab902b36-eaac-4769-b6df-e8ea4d67ffad None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:48.566327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:48.591156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58cbf57-90d7-4e17-8607-00c3797a4ecc None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:48.623467 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fe757c-ee01-4db6-919b-1c4cdbc97218 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:48.624066 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fe757c-ee01-4db6-919b-1c4cdbc97218 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:48.624343 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fe757c-ee01-4db6-919b-1c4cdbc97218 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:48.624615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fe757c-ee01-4db6-919b-1c4cdbc97218 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:48.625273 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fe757c-ee01-4db6-919b-1c4cdbc97218 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:48.625366 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab902b36-eaac-4769-b6df-e8ea4d67ffad None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:48.625542 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fe757c-ee01-4db6-919b-1c4cdbc97218 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:48.626921 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fe757c-ee01-4db6-919b-1c4cdbc97218 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:48.638411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 263/518] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:48.639225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fe757c-ee01-4db6-919b-1c4cdbc97218 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:48.659095 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1ebb57-601d-4cc9-b42a-96ff371f1db8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:48.659714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1ebb57-601d-4cc9-b42a-96ff371f1db8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:48.659988 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1ebb57-601d-4cc9-b42a-96ff371f1db8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:48.660254 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1ebb57-601d-4cc9-b42a-96ff371f1db8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:48.666682 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1ebb57-601d-4cc9-b42a-96ff371f1db8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:48.668144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1ebb57-601d-4cc9-b42a-96ff371f1db8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:48.679998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-aa1ebb57-601d-4cc9-b42a-96ff371f1db8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:05:48.679998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:48.679998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:48.679998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:48.679998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:48.679998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:48.679998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:48.679998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:48.679998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:48.679998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:48.679998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:48.679998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:48.679998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:05:48.679998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:05:48.679998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:05:48.680828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:05:48.680828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:48.680828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:48.680828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:48.680828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:48.680828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:48.680828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:48.680828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:48.680828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:48.680828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:48.680828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:48.680828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:48.680828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:48.680828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:48.680828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:48.680828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:48.680828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:48.681656 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:05:48.681656 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:05:48.681656 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:05:48.681656 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:05:48.681656 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:05:48.681656 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:05:48.681656 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:05:48.681656 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:48.681656 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 256/519] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:48.682114 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1ebb57-601d-4cc9-b42a-96ff371f1db8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:48.684512 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:48.739059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fe757c-ee01-4db6-919b-1c4cdbc97218 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:48.771862 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c22e43-8f8c-4fd0-b999-342e3ef6ccf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:48.772461 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c22e43-8f8c-4fd0-b999-342e3ef6ccf5 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:48.772734 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c22e43-8f8c-4fd0-b999-342e3ef6ccf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:48.773503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c22e43-8f8c-4fd0-b999-342e3ef6ccf5 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:48.774157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c22e43-8f8c-4fd0-b999-342e3ef6ccf5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:48.774453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c22e43-8f8c-4fd0-b999-342e3ef6ccf5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:48.775835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c22e43-8f8c-4fd0-b999-342e3ef6ccf5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:48.777215 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:48.781233 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1ebb57-601d-4cc9-b42a-96ff371f1db8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:48.787820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 264/520] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:48.788609 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c22e43-8f8c-4fd0-b999-342e3ef6ccf5 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:48.816851 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6067ba57-217b-4b25-9058-fab99b2c925f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:48.817469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6067ba57-217b-4b25-9058-fab99b2c925f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:48.817742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6067ba57-217b-4b25-9058-fab99b2c925f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:48.818011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6067ba57-217b-4b25-9058-fab99b2c925f None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:48.825409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6067ba57-217b-4b25-9058-fab99b2c925f None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:48.826830 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6067ba57-217b-4b25-9058-fab99b2c925f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:48.838185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-6067ba57-217b-4b25-9058-fab99b2c925f None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 13 01:05:48.838185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:48.838185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:48.838185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:48.838185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:48.838185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:48.838185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:48.838185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:48.838185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:48.838185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:48.838185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:48.838185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:48.838185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:05:48.838185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:05:48.838185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:05:48.838185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:05:48.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:48.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:48.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:05:48.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:48.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:05:48.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:48.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:48.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:48.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:48.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:48.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:48.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:48.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:48.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:48.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:48.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:48.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:48.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:48.840209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:48.840209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:48.840209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:05:48.840209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:05:48.840209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:05:48.840209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:05:48.840209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:05:48.840209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:05:48.840209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 13 01:05:48.840209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:48.840209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 257/521] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:05:48 2022] GET /identity/v3/users/neutron => generated 85 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:48.840209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6067ba57-217b-4b25-9058-fab99b2c925f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:48.892271 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c22e43-8f8c-4fd0-b999-342e3ef6ccf5 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:48.900969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:48.926104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1601736-b987-4125-9bd0-b10cecf8f718 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:48.926712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1601736-b987-4125-9bd0-b10cecf8f718 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:48.927023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1601736-b987-4125-9bd0-b10cecf8f718 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:48.927292 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1601736-b987-4125-9bd0-b10cecf8f718 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:48.928008 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1601736-b987-4125-9bd0-b10cecf8f718 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:48.928311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1601736-b987-4125-9bd0-b10cecf8f718 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:48.929752 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1601736-b987-4125-9bd0-b10cecf8f718 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:48.938784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6067ba57-217b-4b25-9058-fab99b2c925f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:48.947351 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 265/522] 10.222.0.15 () {56 vars in 1137 bytes} [Tue Sep 13 01:05:48 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:48.948169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1601736-b987-4125-9bd0-b10cecf8f718 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:48.978987 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527dac39-4baf-4bd6-9601-dabfa65ff6ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:48.979592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527dac39-4baf-4bd6-9601-dabfa65ff6ca None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:48.979870 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527dac39-4baf-4bd6-9601-dabfa65ff6ca None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:48.980138 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527dac39-4baf-4bd6-9601-dabfa65ff6ca None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:48.980765 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527dac39-4baf-4bd6-9601-dabfa65ff6ca None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:48.982192 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527dac39-4baf-4bd6-9601-dabfa65ff6ca None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:49.003352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 258/523] 10.222.0.15 () {60 vars in 1132 bytes} [Tue Sep 13 01:05:48 2022] POST /identity/v3/users => generated 247 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:05:49.004104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527dac39-4baf-4bd6-9601-dabfa65ff6ca None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:49.006308 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:49.068032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1601736-b987-4125-9bd0-b10cecf8f718 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:49.106448 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6479ea-c82b-4a90-b85f-9e44227ce47e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:49.107098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6479ea-c82b-4a90-b85f-9e44227ce47e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:49.107376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6479ea-c82b-4a90-b85f-9e44227ce47e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:49.107657 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6479ea-c82b-4a90-b85f-9e44227ce47e None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:49.108329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6479ea-c82b-4a90-b85f-9e44227ce47e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:49.108632 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6479ea-c82b-4a90-b85f-9e44227ce47e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:49.110110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6479ea-c82b-4a90-b85f-9e44227ce47e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:49.120804 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527dac39-4baf-4bd6-9601-dabfa65ff6ca None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:49.133324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 266/524] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:05:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:49.135129 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6479ea-c82b-4a90-b85f-9e44227ce47e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:49.141491 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:49.160960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4ad7a2-144f-4834-aa3a-44426b17b4da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:49.161613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4ad7a2-144f-4834-aa3a-44426b17b4da None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:49.161854 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4ad7a2-144f-4834-aa3a-44426b17b4da None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:49.162140 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4ad7a2-144f-4834-aa3a-44426b17b4da None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:49.171105 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4ad7a2-144f-4834-aa3a-44426b17b4da None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:49.172623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4ad7a2-144f-4834-aa3a-44426b17b4da None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:49.183974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-da4ad7a2-144f-4834-aa3a-44426b17b4da None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:05:49.183974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:49.183974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:49.183974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:49.183974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:49.183974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:49.183974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:49.183974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:49.183974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:49.183974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:49.183974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:49.183974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:49.183974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:05:49.183974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:05:49.183974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:05:49.184813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:05:49.184813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:49.184813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:49.184813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:49.184813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:49.184813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:49.184813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:49.184813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:49.184813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:49.184813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:49.184813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:49.184813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:49.184813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:49.184813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:49.184813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:49.184813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:49.184813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:49.186070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:05:49.186070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:05:49.186070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:05:49.186070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:05:49.186070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:05:49.186070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:05:49.186070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:05:49.186070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:49.186070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 259/525] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:49.186070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4ad7a2-144f-4834-aa3a-44426b17b4da None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:49.265931 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6479ea-c82b-4a90-b85f-9e44227ce47e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:49.281705 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:49.303152 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4ad7a2-144f-4834-aa3a-44426b17b4da None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:49.306243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556ac053-d22b-40ec-9e82-a86700ce863d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:49.306879 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556ac053-d22b-40ec-9e82-a86700ce863d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:49.307174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556ac053-d22b-40ec-9e82-a86700ce863d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:49.307443 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556ac053-d22b-40ec-9e82-a86700ce863d None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:49.308160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556ac053-d22b-40ec-9e82-a86700ce863d None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:49.308460 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556ac053-d22b-40ec-9e82-a86700ce863d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:49.309882 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556ac053-d22b-40ec-9e82-a86700ce863d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:49.333080 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 267/526] 10.222.0.15 () {56 vars in 1139 bytes} [Tue Sep 13 01:05:49 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:49.333080 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556ac053-d22b-40ec-9e82-a86700ce863d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:49.343478 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718ad324-7bb2-4acf-b095-d5cd862d45d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:49.344216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718ad324-7bb2-4acf-b095-d5cd862d45d1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:49.344392 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718ad324-7bb2-4acf-b095-d5cd862d45d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:49.344665 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718ad324-7bb2-4acf-b095-d5cd862d45d1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:49.393053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:49.416724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718ad324-7bb2-4acf-b095-d5cd862d45d1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:49.417688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718ad324-7bb2-4acf-b095-d5cd862d45d1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:49.449155 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556ac053-d22b-40ec-9e82-a86700ce863d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:49.484677 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61b9593-66d9-4997-8213-e8a9c60ac49c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:49.485309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61b9593-66d9-4997-8213-e8a9c60ac49c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:49.485610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61b9593-66d9-4997-8213-e8a9c60ac49c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:49.485921 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61b9593-66d9-4997-8213-e8a9c60ac49c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:49.486713 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61b9593-66d9-4997-8213-e8a9c60ac49c None admin] RBAC: Adding query filter params (role.id=71b1e7c4b5af44ddbb7a517760ad0119, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=2f2da4d2d3604d16b58dc25b8c17c49a) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:49.487083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61b9593-66d9-4997-8213-e8a9c60ac49c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:49.488654 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61b9593-66d9-4997-8213-e8a9c60ac49c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:49.494577 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 268/527] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:05:49 2022] GET /identity/v3/role_assignments?user.id=2f2da4d2d3604d16b58dc25b8c17c49a&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=71b1e7c4b5af44ddbb7a517760ad0119 => generated 259 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:49.495399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61b9593-66d9-4997-8213-e8a9c60ac49c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:49.534197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 260/528] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:49.535550 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718ad324-7bb2-4acf-b095-d5cd862d45d1 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:49.547758 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:49.627972 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61b9593-66d9-4997-8213-e8a9c60ac49c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:49.643257 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:49.657908 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718ad324-7bb2-4acf-b095-d5cd862d45d1 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:49.666620 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce9bd4b-0a5f-4aab-965b-c5a4ab2e847c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:49.667591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce9bd4b-0a5f-4aab-965b-c5a4ab2e847c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:49.668051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce9bd4b-0a5f-4aab-965b-c5a4ab2e847c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:49.668494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce9bd4b-0a5f-4aab-965b-c5a4ab2e847c None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:49.669410 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce9bd4b-0a5f-4aab-965b-c5a4ab2e847c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:49.669899 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce9bd4b-0a5f-4aab-965b-c5a4ab2e847c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:49.671765 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce9bd4b-0a5f-4aab-965b-c5a4ab2e847c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:49.685522 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 269/529] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:05:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:49.686652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce9bd4b-0a5f-4aab-965b-c5a4ab2e847c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:49.705148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750a9837-0764-4441-bb65-b018c2036db5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:49.705870 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750a9837-0764-4441-bb65-b018c2036db5 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:49.706243 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750a9837-0764-4441-bb65-b018c2036db5 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:49.706563 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750a9837-0764-4441-bb65-b018c2036db5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:49.804953 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750a9837-0764-4441-bb65-b018c2036db5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:49.807801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:49.808012 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750a9837-0764-4441-bb65-b018c2036db5 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:49.817095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce9bd4b-0a5f-4aab-965b-c5a4ab2e847c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:49.864929 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5584dc-fbfe-41f8-a368-b03812efade6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:49.865612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5584dc-fbfe-41f8-a368-b03812efade6 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:49.865890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5584dc-fbfe-41f8-a368-b03812efade6 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:49.866160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5584dc-fbfe-41f8-a368-b03812efade6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:49.879979 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5584dc-fbfe-41f8-a368-b03812efade6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:49.881692 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5584dc-fbfe-41f8-a368-b03812efade6 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:49.902054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-9f5584dc-fbfe-41f8-a368-b03812efade6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:49.902054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:49.902054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:49.902054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:49.902054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:49.902054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:49.902054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:49.902054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:49.902054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:49.902054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:49.902054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:49.902054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:49.902054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:49.902054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:49.902054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:49.902054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:49.903083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:49.903083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:49.903083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:49.903083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:49.903083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:49.903083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:49.903083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:49.903083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:49.903083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:49.903083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:49.903083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:49.903083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:49.903083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:49.903083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:49.903083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:49.903083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:49.903083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:49.903915 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:49.903915 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:49.903915 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:49.906929 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 270/530] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:49.907913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5584dc-fbfe-41f8-a368-b03812efade6 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:49.912393 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:49.944087 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 261/531] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:49.944087 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750a9837-0764-4441-bb65-b018c2036db5 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:50.025884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5584dc-fbfe-41f8-a368-b03812efade6 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:50.045625 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750a9837-0764-4441-bb65-b018c2036db5 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:50.062163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67894727-57bd-4a77-a2d3-6f95a5c14054 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:50.063426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67894727-57bd-4a77-a2d3-6f95a5c14054 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:50.063859 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67894727-57bd-4a77-a2d3-6f95a5c14054 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:50.064268 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67894727-57bd-4a77-a2d3-6f95a5c14054 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:50.065118 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67894727-57bd-4a77-a2d3-6f95a5c14054 None admin] RBAC: Adding query filter params (role.id=71b1e7c4b5af44ddbb7a517760ad0119, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=eef98bcf2699461d9a7f3c3bffe82f34) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:50.065573 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67894727-57bd-4a77-a2d3-6f95a5c14054 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:50.067249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67894727-57bd-4a77-a2d3-6f95a5c14054 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:50.073027 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 271/532] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:05:49 2022] GET /identity/v3/role_assignments?user.id=eef98bcf2699461d9a7f3c3bffe82f34&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=71b1e7c4b5af44ddbb7a517760ad0119 => generated 259 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:50.073949 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67894727-57bd-4a77-a2d3-6f95a5c14054 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:50.081866 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b457c6e-f249-463d-9775-1af3270a907e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:50.082495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b457c6e-f249-463d-9775-1af3270a907e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:50.082770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b457c6e-f249-463d-9775-1af3270a907e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:50.083102 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b457c6e-f249-463d-9775-1af3270a907e None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:50.083740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b457c6e-f249-463d-9775-1af3270a907e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:50.084038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b457c6e-f249-463d-9775-1af3270a907e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:50.085995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b457c6e-f249-463d-9775-1af3270a907e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:50.091291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:50.103110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 262/533] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:50.104417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b457c6e-f249-463d-9775-1af3270a907e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:50.150943 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:50.207729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67894727-57bd-4a77-a2d3-6f95a5c14054 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:50.220860 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b457c6e-f249-463d-9775-1af3270a907e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:50.267982 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34811970-d681-4454-a82c-d53e03b97d31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:50.268635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34811970-d681-4454-a82c-d53e03b97d31 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:50.268910 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34811970-d681-4454-a82c-d53e03b97d31 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:50.269179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34811970-d681-4454-a82c-d53e03b97d31 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:50.281100 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ac4b6d-16d9-410b-83c8-679ad0eb0e68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:50.281755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ac4b6d-16d9-410b-83c8-679ad0eb0e68 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:50.282029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ac4b6d-16d9-410b-83c8-679ad0eb0e68 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:50.282295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ac4b6d-16d9-410b-83c8-679ad0eb0e68 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:50.283566 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34811970-d681-4454-a82c-d53e03b97d31 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:50.285042 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34811970-d681-4454-a82c-d53e03b97d31 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:50.297355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ac4b6d-16d9-410b-83c8-679ad0eb0e68 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:50.298787 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ac4b6d-16d9-410b-83c8-679ad0eb0e68 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:50.306626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-34811970-d681-4454-a82c-d53e03b97d31 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:05:50.306626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:50.306626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:50.306626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:50.306626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:50.306626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:50.306626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:50.306626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:50.306626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:50.306626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:50.306626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:50.306626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:50.306626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:05:50.306626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:05:50.306626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:05:50.307526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:05:50.307526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:50.307526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:50.307526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:50.307526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:50.307526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:50.307526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:50.307526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:50.307526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:50.307526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:50.307526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:50.307526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:50.307526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:50.307526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:50.307526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:50.307526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:50.307526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:50.308347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:05:50.308347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:05:50.308347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:05:50.308347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:05:50.308347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:05:50.308347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:05:50.308347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:05:50.308347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:50.310308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 263/534] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:50.315200 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34811970-d681-4454-a82c-d53e03b97d31 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:50.320702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-61ac4b6d-16d9-410b-83c8-679ad0eb0e68 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:50.320702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:50.320702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:50.320702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:50.320702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:50.320702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:50.320702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:50.320702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:50.320702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:50.320702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:50.320702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:50.320702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:50.320702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:50.320702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:50.320702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:50.320702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:50.321769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:50.321769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:50.321769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:50.321769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:50.321769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:50.321769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:50.321769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:50.321769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:50.321769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:50.321769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:50.321769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:50.321769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:50.321769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:50.321769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:50.321769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:50.321769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:50.321769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:50.322808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:50.322808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:50.322808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:50.327011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 272/535] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:50.329179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ac4b6d-16d9-410b-83c8-679ad0eb0e68 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:50.439249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:50.468409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:50.472952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34811970-d681-4454-a82c-d53e03b97d31 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:50.510699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe1c52d-3d21-4948-a82e-541879febac0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:50.511409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe1c52d-3d21-4948-a82e-541879febac0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:50.511740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe1c52d-3d21-4948-a82e-541879febac0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:50.512015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe1c52d-3d21-4948-a82e-541879febac0 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:50.512693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe1c52d-3d21-4948-a82e-541879febac0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:50.513055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe1c52d-3d21-4948-a82e-541879febac0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:50.514489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe1c52d-3d21-4948-a82e-541879febac0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:50.516231 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ac4b6d-16d9-410b-83c8-679ad0eb0e68 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:50.537565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 264/536] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:05:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:50.539496 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe1c52d-3d21-4948-a82e-541879febac0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:50.573515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e57b60-23a7-40dd-b549-0c554bc0fb12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:50.574042 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e57b60-23a7-40dd-b549-0c554bc0fb12 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:50.574319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e57b60-23a7-40dd-b549-0c554bc0fb12 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:50.574585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e57b60-23a7-40dd-b549-0c554bc0fb12 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:50.575241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e57b60-23a7-40dd-b549-0c554bc0fb12 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:50.575538 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e57b60-23a7-40dd-b549-0c554bc0fb12 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:50.576885 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e57b60-23a7-40dd-b549-0c554bc0fb12 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:50.600682 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 273/537] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:50.609735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e57b60-23a7-40dd-b549-0c554bc0fb12 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:50.743898 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe1c52d-3d21-4948-a82e-541879febac0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:50.774561 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e57b60-23a7-40dd-b549-0c554bc0fb12 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:50.800501 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6a0925-d625-491f-93fb-0a5b51cf3b70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:50.801110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6a0925-d625-491f-93fb-0a5b51cf3b70 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:50.801387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6a0925-d625-491f-93fb-0a5b51cf3b70 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:50.801653 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6a0925-d625-491f-93fb-0a5b51cf3b70 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:50.802369 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6a0925-d625-491f-93fb-0a5b51cf3b70 None admin] RBAC: Adding query filter params (role.id=71b1e7c4b5af44ddbb7a517760ad0119, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=f90a5e57daa643eab94c01e21fcbc4c4) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:50.802672 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6a0925-d625-491f-93fb-0a5b51cf3b70 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:50.804917 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6a0925-d625-491f-93fb-0a5b51cf3b70 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:50.817957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 265/538] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:05:50 2022] GET /identity/v3/role_assignments?user.id=f90a5e57daa643eab94c01e21fcbc4c4&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=71b1e7c4b5af44ddbb7a517760ad0119 => generated 259 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:50.826173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:50.845610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0e4626-afcc-407a-b51a-dbe903b5a89b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:50.846254 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0e4626-afcc-407a-b51a-dbe903b5a89b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:50.846529 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0e4626-afcc-407a-b51a-dbe903b5a89b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:50.846795 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0e4626-afcc-407a-b51a-dbe903b5a89b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:50.859105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0e4626-afcc-407a-b51a-dbe903b5a89b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:50.860543 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0e4626-afcc-407a-b51a-dbe903b5a89b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:50.876957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-4c0e4626-afcc-407a-b51a-dbe903b5a89b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:05:50.876957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:50.876957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:50.876957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:50.876957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:50.876957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:50.876957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:50.876957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:50.876957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:50.876957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:50.876957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:50.876957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:50.876957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:05:50.876957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:05:50.876957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:05:50.878013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:05:50.878013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:50.878013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:50.878013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:50.878013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:50.878013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:50.878013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:50.878013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:50.878013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:50.878013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:50.878013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:50.878013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:50.878013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:50.878013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:50.878013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:50.878013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:50.878013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:50.879451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:05:50.879451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:05:50.879451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:05:50.879451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:05:50.879451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:05:50.879451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:05:50.879451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:05:50.879451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:50.880105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 274/539] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:50.884515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6a0925-d625-491f-93fb-0a5b51cf3b70 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:50.922193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:51.058715 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6a0925-d625-491f-93fb-0a5b51cf3b70 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:51.113682 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f3dfc9-a060-4051-96b8-76156c2b523b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:51.114230 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f3dfc9-a060-4051-96b8-76156c2b523b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:51.114511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f3dfc9-a060-4051-96b8-76156c2b523b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:51.114785 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f3dfc9-a060-4051-96b8-76156c2b523b None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:51.115511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f3dfc9-a060-4051-96b8-76156c2b523b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:51.115818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f3dfc9-a060-4051-96b8-76156c2b523b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:51.117347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f3dfc9-a060-4051-96b8-76156c2b523b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:51.135173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 266/540] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:05:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:51.140821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0e4626-afcc-407a-b51a-dbe903b5a89b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:51.312302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0e4626-afcc-407a-b51a-dbe903b5a89b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:51.355870 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b4ca5c-63f4-4bb9-b126-bb3a8d41c65d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:51.356474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b4ca5c-63f4-4bb9-b126-bb3a8d41c65d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:51.356749 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b4ca5c-63f4-4bb9-b126-bb3a8d41c65d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:51.357016 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b4ca5c-63f4-4bb9-b126-bb3a8d41c65d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:51.357729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b4ca5c-63f4-4bb9-b126-bb3a8d41c65d None admin] RBAC: Adding query filter params (role.id=71b1e7c4b5af44ddbb7a517760ad0119, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=09cd263f72484733adaaa0850ebfa609) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:51.358030 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b4ca5c-63f4-4bb9-b126-bb3a8d41c65d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:51.359766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b4ca5c-63f4-4bb9-b126-bb3a8d41c65d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:51.372018 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 275/541] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:05:51 2022] GET /identity/v3/role_assignments?user.id=09cd263f72484733adaaa0850ebfa609&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=71b1e7c4b5af44ddbb7a517760ad0119 => generated 259 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:51.456421 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9760288d-914c-4998-952c-fafe0cc839db None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:51.457049 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9760288d-914c-4998-952c-fafe0cc839db None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:51.457049 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9760288d-914c-4998-952c-fafe0cc839db None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:51.458519 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 267/542] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:51.464744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ac5a3e-d17b-4d9e-974c-8ad911285485 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:51.465108 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ac5a3e-d17b-4d9e-974c-8ad911285485 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:51.465344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ac5a3e-d17b-4d9e-974c-8ad911285485 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:51.496194 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-28ac5a3e-d17b-4d9e-974c-8ad911285485 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:51.497411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-28ac5a3e-d17b-4d9e-974c-8ad911285485 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:51.564744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:51.602806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 276/543] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:52.020660 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9760288d-914c-4998-952c-fafe0cc839db None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:52.129663 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:52.160090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9760288d-914c-4998-952c-fafe0cc839db None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:52.205433 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8e84fe-8118-4011-ac43-03bcc9c3c80a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:52.206085 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8e84fe-8118-4011-ac43-03bcc9c3c80a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:52.206359 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8e84fe-8118-4011-ac43-03bcc9c3c80a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:52.206629 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8e84fe-8118-4011-ac43-03bcc9c3c80a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:52.222912 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca8e84fe-8118-4011-ac43-03bcc9c3c80a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:52.227994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-ca8e84fe-8118-4011-ac43-03bcc9c3c80a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:52.227994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:52.227994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:52.227994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:52.227994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:52.227994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:52.227994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:52.227994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:52.227994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:52.227994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:52.227994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:52.227994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:52.227994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:52.227994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:52.227994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:52.227994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:52.230172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:52.230172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:52.230172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:52.230172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:52.230172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:52.230172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:52.230172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:52.230172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:52.230172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:52.230172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:52.230172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:52.230172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:52.230172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:52.230172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:52.230172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:52.230172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:52.232201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:52.232201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:52.232201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:52.232201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:52.232201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:52.232201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:52.232201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:52.232201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:52.232201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:52.232201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:52.232201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:52.232201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:52.232201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 268/544] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:05:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:52.238877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ac5a3e-d17b-4d9e-974c-8ad911285485 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:52.364946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ac5a3e-d17b-4d9e-974c-8ad911285485 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:52.412148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b7b179-6ad6-43a4-9bda-64e81bc58038 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:52.412690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b7b179-6ad6-43a4-9bda-64e81bc58038 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:52.413168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b7b179-6ad6-43a4-9bda-64e81bc58038 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:52.415770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 269/545] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:52 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:05:52.421963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eab3c42-ecd8-4392-9708-b53364565b9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:52.422489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1e0604-3515-45e4-a5f2-d1b9478cd9a7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:52.422603 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eab3c42-ecd8-4392-9708-b53364565b9f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:52.422877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eab3c42-ecd8-4392-9708-b53364565b9f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:52.423153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eab3c42-ecd8-4392-9708-b53364565b9f None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:52.423504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1e0604-3515-45e4-a5f2-d1b9478cd9a7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:52.423821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eab3c42-ecd8-4392-9708-b53364565b9f None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:52.424122 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eab3c42-ecd8-4392-9708-b53364565b9f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:52.424363 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1e0604-3515-45e4-a5f2-d1b9478cd9a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:52.425481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eab3c42-ecd8-4392-9708-b53364565b9f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:52.445149 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6c1e0604-3515-45e4-a5f2-d1b9478cd9a7 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:52.446239 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 277/546] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:05:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:52.447013 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6c1e0604-3515-45e4-a5f2-d1b9478cd9a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:52.452062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eab3c42-ecd8-4392-9708-b53364565b9f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:52.540567 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:52.576481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 270/547] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 01:05:52.614881 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7320bad2-8ab1-4713-9cb2-d17e5073c364 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:52.615220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7320bad2-8ab1-4713-9cb2-d17e5073c364 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:52.615485 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7320bad2-8ab1-4713-9cb2-d17e5073c364 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:52.617133 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 271/548] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:05:52.629323 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9321f9-9f8e-4829-896f-d02a8911d3fa None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:52.629609 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9321f9-9f8e-4829-896f-d02a8911d3fa None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:52.629884 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9321f9-9f8e-4829-896f-d02a8911d3fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:52.638167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eab3c42-ecd8-4392-9708-b53364565b9f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:52.661447 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-af9321f9-9f8e-4829-896f-d02a8911d3fa None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:52.662639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-af9321f9-9f8e-4829-896f-d02a8911d3fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:52.682268 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461d707e-425d-4f6b-95bc-3f40d7a4a637 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:52.682921 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461d707e-425d-4f6b-95bc-3f40d7a4a637 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:52.683200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461d707e-425d-4f6b-95bc-3f40d7a4a637 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:52.683471 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461d707e-425d-4f6b-95bc-3f40d7a4a637 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:52.705049 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461d707e-425d-4f6b-95bc-3f40d7a4a637 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:52.706488 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461d707e-425d-4f6b-95bc-3f40d7a4a637 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:52.731855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-461d707e-425d-4f6b-95bc-3f40d7a4a637 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:52.731855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:52.731855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:52.731855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:52.731855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:52.731855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:52.731855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:52.731855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:52.731855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:52.731855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:52.731855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:52.731855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:52.731855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:52.731855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:52.731855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:52.731855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:52.732763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:52.732763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:52.732763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:52.732763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:52.732763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:52.732763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:52.732763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:52.732763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:52.732763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:52.732763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:52.732763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:52.732763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:52.732763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:52.732763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:52.732763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:52.732763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:52.732763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:52.734400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:52.734400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:52.734400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:52.734400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 278/549] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:05:52.735674 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 272/550] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:52.738537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461d707e-425d-4f6b-95bc-3f40d7a4a637 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:52.880676 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:52.892591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461d707e-425d-4f6b-95bc-3f40d7a4a637 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:52.902167 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9321f9-9f8e-4829-896f-d02a8911d3fa None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:52.904808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:52.947721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7267d47c-0a48-4292-a8bd-32a9728877ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:52.948335 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7267d47c-0a48-4292-a8bd-32a9728877ed None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:52.948612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7267d47c-0a48-4292-a8bd-32a9728877ed None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:52.948884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7267d47c-0a48-4292-a8bd-32a9728877ed None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:52.949521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7267d47c-0a48-4292-a8bd-32a9728877ed None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:52.949823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7267d47c-0a48-4292-a8bd-32a9728877ed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:52.951203 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7267d47c-0a48-4292-a8bd-32a9728877ed None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:52.978453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 279/551] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:52.985711 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7267d47c-0a48-4292-a8bd-32a9728877ed None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:53.092615 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9321f9-9f8e-4829-896f-d02a8911d3fa None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:53.130456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7267d47c-0a48-4292-a8bd-32a9728877ed None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:53.139494 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bec4d0-80fe-4842-adbd-35bd8a8c1715 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:53.140158 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bec4d0-80fe-4842-adbd-35bd8a8c1715 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:53.140540 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bec4d0-80fe-4842-adbd-35bd8a8c1715 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:53.141276 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bec4d0-80fe-4842-adbd-35bd8a8c1715 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:53.155263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bec4d0-80fe-4842-adbd-35bd8a8c1715 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:53.162441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-b0bec4d0-80fe-4842-adbd-35bd8a8c1715 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:53.162441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:53.162441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:53.162441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:53.162441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:53.162441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:53.162441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:53.162441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:53.162441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:53.162441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:53.162441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:53.162441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:53.162441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:53.162441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:53.162441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:53.162441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:53.164088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:53.164088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:53.164088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:53.164088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:53.164088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:53.164088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:53.164088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:53.164088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:53.164088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:53.164088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:53.164088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:53.164088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:53.164088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:53.164088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:53.164088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:53.164088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:53.165514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:53.165514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:53.165514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:53.165514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:53.165514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:53.165514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:53.165514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:53.165514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:53.165514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:53.165514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:53.165514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:53.165514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:53.165514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 273/552] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:05:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:05:53.166201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bec4d0-80fe-4842-adbd-35bd8a8c1715 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:53.172018 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9871b5dc-b86d-4585-a396-dfb3e57416a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:53.172649 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9871b5dc-b86d-4585-a396-dfb3e57416a7 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:53.172926 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9871b5dc-b86d-4585-a396-dfb3e57416a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:53.173192 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9871b5dc-b86d-4585-a396-dfb3e57416a7 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:53.181125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9871b5dc-b86d-4585-a396-dfb3e57416a7 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:53.182731 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9871b5dc-b86d-4585-a396-dfb3e57416a7 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:53.200670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-9871b5dc-b86d-4585-a396-dfb3e57416a7 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 13 01:05:53.200670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:53.200670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:53.200670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:53.200670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:53.200670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:53.200670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:53.200670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:53.200670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:53.200670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:53.200670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:53.200670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:53.200670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:05:53.200670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:05:53.200670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:05:53.200670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:05:53.202551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:53.202551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:53.202551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:05:53.202551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:53.202551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:05:53.202551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:53.202551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:53.202551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:53.202551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:53.202551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:53.202551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:53.202551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:53.202551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:53.202551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:53.202551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:53.202551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:53.202551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:53.202551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:53.204970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:53.204970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:53.204970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:05:53.204970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:05:53.204970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:05:53.204970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:05:53.204970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:05:53.204970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:05:53.204970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 13 01:05:53.204970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:53.204970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 280/553] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:05:52 2022] GET /identity/v3/users/swift => generated 83 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:05:53.204970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9871b5dc-b86d-4585-a396-dfb3e57416a7 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:53.219286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:53.266970 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:53.318298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9871b5dc-b86d-4585-a396-dfb3e57416a7 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:53.325153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bec4d0-80fe-4842-adbd-35bd8a8c1715 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:53.353972 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1f5a6b-1dd2-421f-b472-a09115888d34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:53.354728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1f5a6b-1dd2-421f-b472-a09115888d34 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:53.355137 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1f5a6b-1dd2-421f-b472-a09115888d34 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:53.355658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1f5a6b-1dd2-421f-b472-a09115888d34 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:53.356522 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1f5a6b-1dd2-421f-b472-a09115888d34 None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:53.357011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1f5a6b-1dd2-421f-b472-a09115888d34 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:53.358495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1f5a6b-1dd2-421f-b472-a09115888d34 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:53.367624 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 281/554] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:05:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:53.368603 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1f5a6b-1dd2-421f-b472-a09115888d34 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:53.375426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78de5a05-0bd1-4c2a-859f-f91778375030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:53.376092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78de5a05-0bd1-4c2a-859f-f91778375030 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:53.376375 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78de5a05-0bd1-4c2a-859f-f91778375030 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:53.376650 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78de5a05-0bd1-4c2a-859f-f91778375030 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:53.393112 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78de5a05-0bd1-4c2a-859f-f91778375030 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:53.400260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-78de5a05-0bd1-4c2a-859f-f91778375030 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:53.400260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:53.400260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:53.400260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:53.400260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:53.400260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:53.400260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:53.400260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:53.400260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:53.400260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:53.400260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:53.400260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:53.400260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:53.400260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:53.400260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:53.400260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:53.401882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:53.401882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:53.401882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:53.401882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:53.401882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:53.401882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:53.401882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:53.401882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:53.401882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:53.401882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:53.401882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:53.401882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:53.401882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:53.401882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:53.401882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:53.401882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:53.419611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:53.419611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:53.419611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:53.419611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:53.419611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:53.419611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:53.419611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:53.419611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:53.419611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:53.419611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:53.419611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:53.419611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:53.419611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 274/555] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:05:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:53.419611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78de5a05-0bd1-4c2a-859f-f91778375030 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:53.465753 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:53.506320 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1f5a6b-1dd2-421f-b472-a09115888d34 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:53.528763 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78de5a05-0bd1-4c2a-859f-f91778375030 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:53.543396 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2a2d73-798f-4add-acee-10ff4b658bf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:53.544006 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2a2d73-798f-4add-acee-10ff4b658bf0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:53.544289 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2a2d73-798f-4add-acee-10ff4b658bf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:53.544568 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2a2d73-798f-4add-acee-10ff4b658bf0 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:53.545290 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2a2d73-798f-4add-acee-10ff4b658bf0 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:53.545598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2a2d73-798f-4add-acee-10ff4b658bf0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:53.547257 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2a2d73-798f-4add-acee-10ff4b658bf0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:53.552272 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:53.565152 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23dd86a7-b61d-4007-be90-e020a2ee758e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:53.565772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dd86a7-b61d-4007-be90-e020a2ee758e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:53.566049 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dd86a7-b61d-4007-be90-e020a2ee758e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:53.566315 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dd86a7-b61d-4007-be90-e020a2ee758e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:53.572471 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 282/556] 10.222.0.15 () {56 vars in 1135 bytes} [Tue Sep 13 01:05:53 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:53.573211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2a2d73-798f-4add-acee-10ff4b658bf0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:53.577194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23dd86a7-b61d-4007-be90-e020a2ee758e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:53.578635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23dd86a7-b61d-4007-be90-e020a2ee758e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:53.593226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-23dd86a7-b61d-4007-be90-e020a2ee758e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:53.593226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:53.593226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:53.593226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:53.593226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:53.593226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:53.593226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:53.593226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:53.593226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:53.593226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:53.593226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:53.593226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:53.593226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:53.593226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:53.593226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:53.593226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:53.594780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:53.594780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:53.594780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:53.594780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:53.594780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:53.594780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:53.594780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:53.594780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:53.594780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:53.594780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:53.594780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:53.594780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:53.594780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:53.594780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:53.594780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:53.594780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:53.594780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:53.595709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:53.595709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:53.595709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:53.595709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 275/557] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:53.595709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8187ddc3-df46-40ca-980b-674a64cea943 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:53.596035 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8187ddc3-df46-40ca-980b-674a64cea943 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:53.596035 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8187ddc3-df46-40ca-980b-674a64cea943 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:53.597464 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 276/558] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:53 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:53.598154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8187ddc3-df46-40ca-980b-674a64cea943 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:53.683443 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2a2d73-798f-4add-acee-10ff4b658bf0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:53.705914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8187ddc3-df46-40ca-980b-674a64cea943 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:53.718421 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:53.726605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db3a03f-8659-4c48-8077-a4522f96d7a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:53.727246 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db3a03f-8659-4c48-8077-a4522f96d7a0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:53.728040 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db3a03f-8659-4c48-8077-a4522f96d7a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:53.728319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db3a03f-8659-4c48-8077-a4522f96d7a0 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:53.729253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db3a03f-8659-4c48-8077-a4522f96d7a0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:53.729562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db3a03f-8659-4c48-8077-a4522f96d7a0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:53.731180 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db3a03f-8659-4c48-8077-a4522f96d7a0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:53.739924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 283/559] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:05:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:53.740948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db3a03f-8659-4c48-8077-a4522f96d7a0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:53.746034 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1191f2-2f88-4c64-b21d-e41764aa68d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:53.746896 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1191f2-2f88-4c64-b21d-e41764aa68d8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:53.747182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1191f2-2f88-4c64-b21d-e41764aa68d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:53.748309 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1191f2-2f88-4c64-b21d-e41764aa68d8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:53.797839 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:53.827442 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1191f2-2f88-4c64-b21d-e41764aa68d8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:53.829256 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1191f2-2f88-4c64-b21d-e41764aa68d8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:53.866880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db3a03f-8659-4c48-8077-a4522f96d7a0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:53.900760 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81337e8e-abd4-4bf7-9b35-5b958c8cc16b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:53.901375 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81337e8e-abd4-4bf7-9b35-5b958c8cc16b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:53.901654 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81337e8e-abd4-4bf7-9b35-5b958c8cc16b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:53.901927 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81337e8e-abd4-4bf7-9b35-5b958c8cc16b None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:53.902564 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81337e8e-abd4-4bf7-9b35-5b958c8cc16b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:53.902894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81337e8e-abd4-4bf7-9b35-5b958c8cc16b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:53.904262 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81337e8e-abd4-4bf7-9b35-5b958c8cc16b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:53.917344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 284/560] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:53.918585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c346343f-2f47-44a2-ae74-a38a91af26ec None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:53.918878 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c346343f-2f47-44a2-ae74-a38a91af26ec None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:53.919173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c346343f-2f47-44a2-ae74-a38a91af26ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:53.935098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 277/561] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:05:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:05:53.935098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1191f2-2f88-4c64-b21d-e41764aa68d8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:53.936433 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c346343f-2f47-44a2-ae74-a38a91af26ec None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:53.937593 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c346343f-2f47-44a2-ae74-a38a91af26ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:53.985622 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:54.005120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 285/562] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:54.006401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc707dc-15b1-41c3-a396-ec774816ef9f None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:54.006683 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc707dc-15b1-41c3-a396-ec774816ef9f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:54.006979 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc707dc-15b1-41c3-a396-ec774816ef9f None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:54.008368 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 286/563] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:54.008961 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc707dc-15b1-41c3-a396-ec774816ef9f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:54.009458 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:54.037261 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1191f2-2f88-4c64-b21d-e41764aa68d8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:54.073643 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e86971-6ed9-4330-bde5-356c9bec1c3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:54.074257 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e86971-6ed9-4330-bde5-356c9bec1c3a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:54.074529 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e86971-6ed9-4330-bde5-356c9bec1c3a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:54.074795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e86971-6ed9-4330-bde5-356c9bec1c3a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:54.084992 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e86971-6ed9-4330-bde5-356c9bec1c3a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:54.087367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e86971-6ed9-4330-bde5-356c9bec1c3a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:54.101544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-c0e86971-6ed9-4330-bde5-356c9bec1c3a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:54.101544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:54.101544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:54.101544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:54.101544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:54.101544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:54.101544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:54.101544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:54.101544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:54.101544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:54.101544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:54.101544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:54.101544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:54.101544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:54.101544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:54.101544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:54.102688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:54.102688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:54.102688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:54.102688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:54.102688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:54.102688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:54.102688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:54.102688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:54.102688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:54.102688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:54.102688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:54.102688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:54.102688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:54.102688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:54.102688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:54.102688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:54.102688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:54.103694 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:54.103694 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:54.103694 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:54.103694 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 278/564] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:54.103924 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e86971-6ed9-4330-bde5-356c9bec1c3a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:54.130288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc707dc-15b1-41c3-a396-ec774816ef9f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:54.164986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b387f2d4-2e31-4f53-a0eb-fa182d6b6aae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:54.165619 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b387f2d4-2e31-4f53-a0eb-fa182d6b6aae None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:54.165898 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b387f2d4-2e31-4f53-a0eb-fa182d6b6aae None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:54.166168 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b387f2d4-2e31-4f53-a0eb-fa182d6b6aae None admin] PATH_INFO: `/v3/users/nova` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:54.173119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b387f2d4-2e31-4f53-a0eb-fa182d6b6aae None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:54.174556 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b387f2d4-2e31-4f53-a0eb-fa182d6b6aae None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:54.186160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-b387f2d4-2e31-4f53-a0eb-fa182d6b6aae None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 01:05:54.186160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:54.186160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:54.186160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:54.186160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:54.186160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:54.186160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:54.186160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:54.186160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:54.186160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:54.186160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:54.186160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:54.186160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:05:54.186160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:05:54.186160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:05:54.186160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:05:54.187153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:54.187153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:54.187153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:05:54.187153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:54.187153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:05:54.187153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:54.187153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:54.187153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:54.187153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:54.187153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:54.187153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:54.187153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:54.187153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:54.187153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:54.187153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:54.187153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:54.187153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:54.187153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:54.188697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:54.188697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:54.188697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:05:54.188697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:05:54.188697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:05:54.188697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:05:54.188697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:05:54.188697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:05:54.188697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 01:05:54.188697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:54.191974 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 287/565] 10.222.0.15 () {56 vars in 1093 bytes} [Tue Sep 13 01:05:54 2022] GET /identity/v3/users/nova => generated 82 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:54.193187 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2437dd78-5979-4933-be4c-e99986d22ebb None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:54.193472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2437dd78-5979-4933-be4c-e99986d22ebb None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:54.193733 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2437dd78-5979-4933-be4c-e99986d22ebb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:54.212210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-2437dd78-5979-4933-be4c-e99986d22ebb None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:54.213393 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2437dd78-5979-4933-be4c-e99986d22ebb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:54.215593 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e86971-6ed9-4330-bde5-356c9bec1c3a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:54.241379 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:54.250802 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:54.252312 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66dbde4d-8598-4336-ba6b-43effe86a905 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:54.253016 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66dbde4d-8598-4336-ba6b-43effe86a905 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:54.253303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66dbde4d-8598-4336-ba6b-43effe86a905 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:54.253577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66dbde4d-8598-4336-ba6b-43effe86a905 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:54.265650 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66dbde4d-8598-4336-ba6b-43effe86a905 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:54.267332 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66dbde4d-8598-4336-ba6b-43effe86a905 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:54.282090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-66dbde4d-8598-4336-ba6b-43effe86a905 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:54.282090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:54.282090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:54.282090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:54.282090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:54.282090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:54.282090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:54.282090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:54.282090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:54.282090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:54.282090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:54.282090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:54.282090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:54.282090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:54.282090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:54.282090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:54.283090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:54.283090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:54.283090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:54.283090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:54.283090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:54.283090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:54.283090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:54.283090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:54.283090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:54.283090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:54.283090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:54.283090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:54.283090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:54.283090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:54.283090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:54.283090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:54.283090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:54.283983 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:54.283983 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:54.283983 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:54.283983 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 279/566] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:54.284536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66dbde4d-8598-4336-ba6b-43effe86a905 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:54.284823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 288/567] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:54.285657 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2437dd78-5979-4933-be4c-e99986d22ebb None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:54.388820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2437dd78-5979-4933-be4c-e99986d22ebb None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:54.405749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66dbde4d-8598-4336-ba6b-43effe86a905 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:54.435106 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a67d9a-736a-4cc0-8736-c9f2ac73fa50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:54.435873 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a67d9a-736a-4cc0-8736-c9f2ac73fa50 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:54.436085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a67d9a-736a-4cc0-8736-c9f2ac73fa50 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:54.436413 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a67d9a-736a-4cc0-8736-c9f2ac73fa50 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:54.437261 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a67d9a-736a-4cc0-8736-c9f2ac73fa50 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:54.437369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a67d9a-736a-4cc0-8736-c9f2ac73fa50 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:54.439091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a67d9a-736a-4cc0-8736-c9f2ac73fa50 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:54.442238 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a877325a-6fc7-41ff-ba5f-177a082a0385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:54.442836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a877325a-6fc7-41ff-ba5f-177a082a0385 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:54.443144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a877325a-6fc7-41ff-ba5f-177a082a0385 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:54.443410 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a877325a-6fc7-41ff-ba5f-177a082a0385 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:54.444061 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a877325a-6fc7-41ff-ba5f-177a082a0385 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:54.444360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a877325a-6fc7-41ff-ba5f-177a082a0385 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:54.445729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a877325a-6fc7-41ff-ba5f-177a082a0385 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:54.461869 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 289/568] 10.222.0.15 () {56 vars in 1133 bytes} [Tue Sep 13 01:05:54 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:54.461869 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a67d9a-736a-4cc0-8736-c9f2ac73fa50 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:54.463146 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 280/569] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:54.464103 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a877325a-6fc7-41ff-ba5f-177a082a0385 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:54.492111 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:54.503491 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:54.566153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a877325a-6fc7-41ff-ba5f-177a082a0385 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:54.572617 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a67d9a-736a-4cc0-8736-c9f2ac73fa50 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:54.599620 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b8fc02-9dd9-421f-9b84-e8797f3a2037 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:54.600236 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b8fc02-9dd9-421f-9b84-e8797f3a2037 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:54.600514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b8fc02-9dd9-421f-9b84-e8797f3a2037 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:54.600789 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b8fc02-9dd9-421f-9b84-e8797f3a2037 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:54.601446 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b8fc02-9dd9-421f-9b84-e8797f3a2037 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:54.601750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b8fc02-9dd9-421f-9b84-e8797f3a2037 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:54.603139 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b8fc02-9dd9-421f-9b84-e8797f3a2037 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:54.607612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2eb769-d4be-4171-884e-d1ff21830ab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:54.608230 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2eb769-d4be-4171-884e-d1ff21830ab0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:54.608504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2eb769-d4be-4171-884e-d1ff21830ab0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:54.608770 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2eb769-d4be-4171-884e-d1ff21830ab0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:54.613927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 281/570] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:54.614769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b8fc02-9dd9-421f-9b84-e8797f3a2037 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:54.622629 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2eb769-d4be-4171-884e-d1ff21830ab0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:54.627454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-0d2eb769-d4be-4171-884e-d1ff21830ab0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:54.627454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:54.627454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:54.627454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:54.627454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:54.627454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:54.627454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:54.627454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:54.627454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:54.627454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:54.627454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:54.627454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:54.627454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:54.627454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:54.627454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:54.627454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:54.628424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:54.628424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:54.628424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:54.628424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:54.628424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:54.628424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:54.628424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:54.628424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:54.628424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:54.628424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:54.628424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:54.628424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:54.628424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:54.628424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:54.628424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:54.628424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:54.629255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:54.629255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:54.629255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:54.629255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:54.629255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:54.629255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:54.629255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:54.629255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:54.629255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:54.629255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:54.629255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:54.629255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:54.629255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 290/571] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:05:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:54.629936 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2eb769-d4be-4171-884e-d1ff21830ab0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:54.708722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:54.713011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b8fc02-9dd9-421f-9b84-e8797f3a2037 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:54.720086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:54.730835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2eb769-d4be-4171-884e-d1ff21830ab0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:54.747539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee355781-f607-43ab-81f8-93144f3992a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:54.748172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee355781-f607-43ab-81f8-93144f3992a7 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:54.748447 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee355781-f607-43ab-81f8-93144f3992a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:54.748714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee355781-f607-43ab-81f8-93144f3992a7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:54.758841 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee355781-f607-43ab-81f8-93144f3992a7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:54.760296 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee355781-f607-43ab-81f8-93144f3992a7 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:54.763623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45f9ad4-366f-42e6-8727-b57eebf1aacd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:54.764244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45f9ad4-366f-42e6-8727-b57eebf1aacd None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:54.764520 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45f9ad4-366f-42e6-8727-b57eebf1aacd None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:54.764786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45f9ad4-366f-42e6-8727-b57eebf1aacd None admin] PATH_INFO: `/v3/users/glance` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:54.771632 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45f9ad4-366f-42e6-8727-b57eebf1aacd None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:54.773539 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45f9ad4-366f-42e6-8727-b57eebf1aacd None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:54.774900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-ee355781-f607-43ab-81f8-93144f3992a7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:54.774900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:54.774900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:54.774900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:54.774900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:54.774900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:54.774900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:54.774900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:54.774900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:54.774900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:54.774900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:54.774900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:54.774900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:54.774900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:54.774900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:54.774900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:54.775965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:54.775965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:54.775965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:54.775965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:54.775965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:54.775965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:54.775965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:54.775965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:54.775965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:54.775965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:54.775965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:54.775965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:54.775965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:54.775965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:54.775965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:54.775965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:54.775965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:54.776893 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:54.776893 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:54.776893 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:54.776893 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 282/572] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:54.776893 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee355781-f607-43ab-81f8-93144f3992a7 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:54.787771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-d45f9ad4-366f-42e6-8727-b57eebf1aacd None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 13 01:05:54.787771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:54.787771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:54.787771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:54.787771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:54.787771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:54.787771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:54.787771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:54.787771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:54.787771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:54.787771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:54.787771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:54.787771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:05:54.787771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:05:54.787771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:05:54.787771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:05:54.789379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:54.789379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:54.789379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:05:54.789379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:54.789379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:05:54.789379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:54.789379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:54.789379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:54.789379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:54.789379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:54.789379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:54.789379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:54.789379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:54.789379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:54.789379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:54.789379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:54.789379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:54.789379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:54.790360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:54.790360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:54.790360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:05:54.790360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:05:54.790360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:05:54.790360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:05:54.790360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:05:54.790360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:05:54.790360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 13 01:05:54.790360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:54.790360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 291/573] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:54 2022] GET /identity/v3/users/glance => generated 84 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:54.790360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45f9ad4-366f-42e6-8727-b57eebf1aacd None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:54.877065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee355781-f607-43ab-81f8-93144f3992a7 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:54.890457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45f9ad4-366f-42e6-8727-b57eebf1aacd None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:54.909456 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382800b5-a8ef-4015-8e17-8051dc88f388 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:54.910077 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382800b5-a8ef-4015-8e17-8051dc88f388 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:54.910356 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382800b5-a8ef-4015-8e17-8051dc88f388 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:54.910631 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382800b5-a8ef-4015-8e17-8051dc88f388 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:54.920982 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382800b5-a8ef-4015-8e17-8051dc88f388 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:54.923160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f49882-61c5-4264-9788-c5fa7ab4e4c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:54.923769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f49882-61c5-4264-9788-c5fa7ab4e4c7 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:54.924043 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f49882-61c5-4264-9788-c5fa7ab4e4c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:54.924309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f49882-61c5-4264-9788-c5fa7ab4e4c7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:54.926215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-382800b5-a8ef-4015-8e17-8051dc88f388 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:54.926215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:54.926215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:54.926215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:54.926215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:54.926215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:54.926215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:54.926215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:54.926215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:54.926215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:54.926215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:54.926215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:54.926215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:54.926215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:54.926215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:54.926215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:54.927142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:54.927142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:54.927142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:54.927142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:54.927142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:54.927142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:54.927142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:54.927142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:54.927142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:54.927142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:54.927142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:54.927142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:54.927142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:54.927142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:54.927142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:54.927142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:54.927997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:54.927997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:54.927997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:54.927997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:54.927997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:54.927997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:54.927997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:54.927997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:54.927997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:54.927997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:54.927997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:54.927997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:54.927997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 283/574] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:05:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:54.928665 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382800b5-a8ef-4015-8e17-8051dc88f388 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:54.932764 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f49882-61c5-4264-9788-c5fa7ab4e4c7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:54.934180 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f49882-61c5-4264-9788-c5fa7ab4e4c7 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:54.940739 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:54.946671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-e1f49882-61c5-4264-9788-c5fa7ab4e4c7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:05:54.946671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:54.946671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:54.946671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:54.946671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:54.946671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:54.946671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:54.946671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:54.946671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:54.946671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:54.946671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:54.946671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:54.946671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:05:54.946671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:05:54.946671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:05:54.947591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:05:54.947591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:54.947591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:54.947591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:54.947591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:54.947591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:54.947591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:54.947591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:54.947591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:54.947591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:54.947591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:54.947591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:54.947591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:54.947591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:54.947591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:54.947591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:54.947591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:54.948480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:05:54.948480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:05:54.948480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:05:54.948480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:05:54.948480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:05:54.948480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:05:54.948480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:05:54.948480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:54.948480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 292/575] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:54.948971 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f49882-61c5-4264-9788-c5fa7ab4e4c7 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:54.961931 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:55.033600 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382800b5-a8ef-4015-8e17-8051dc88f388 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:55.050110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f49882-61c5-4264-9788-c5fa7ab4e4c7 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:55.066729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8f2f35-2de5-4c6a-a45b-bce784ac5fd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:55.067366 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8f2f35-2de5-4c6a-a45b-bce784ac5fd3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:55.067648 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8f2f35-2de5-4c6a-a45b-bce784ac5fd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:55.067921 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8f2f35-2de5-4c6a-a45b-bce784ac5fd3 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:55.068576 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8f2f35-2de5-4c6a-a45b-bce784ac5fd3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:55.068881 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8f2f35-2de5-4c6a-a45b-bce784ac5fd3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:55.070879 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8f2f35-2de5-4c6a-a45b-bce784ac5fd3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:55.079031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 284/576] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:05:54 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:55.079804 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8f2f35-2de5-4c6a-a45b-bce784ac5fd3 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:55.084132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7c7372-0aa0-49c0-9935-752121769ca8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:55.084734 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7c7372-0aa0-49c0-9935-752121769ca8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:55.085009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7c7372-0aa0-49c0-9935-752121769ca8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:55.085850 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7c7372-0aa0-49c0-9935-752121769ca8 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:55.086449 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7c7372-0aa0-49c0-9935-752121769ca8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:55.086747 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7c7372-0aa0-49c0-9935-752121769ca8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:55.088125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7c7372-0aa0-49c0-9935-752121769ca8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:55.104687 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 293/577] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:55.104687 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7c7372-0aa0-49c0-9935-752121769ca8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:55.152387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:55.167951 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:55.182652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8f2f35-2de5-4c6a-a45b-bce784ac5fd3 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:55.199098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7c7372-0aa0-49c0-9935-752121769ca8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:55.215854 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642fe74c-c329-4a52-b0ae-1fb16dd473c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:55.216397 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642fe74c-c329-4a52-b0ae-1fb16dd473c3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:55.216677 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642fe74c-c329-4a52-b0ae-1fb16dd473c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:55.216951 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642fe74c-c329-4a52-b0ae-1fb16dd473c3 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:55.217665 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642fe74c-c329-4a52-b0ae-1fb16dd473c3 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:55.217977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642fe74c-c329-4a52-b0ae-1fb16dd473c3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:55.219427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642fe74c-c329-4a52-b0ae-1fb16dd473c3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:55.231502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ae3446-b29d-470b-b4cf-9243fd40874f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:55.232092 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ae3446-b29d-470b-b4cf-9243fd40874f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:55.232367 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ae3446-b29d-470b-b4cf-9243fd40874f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:55.232631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ae3446-b29d-470b-b4cf-9243fd40874f None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:55.233300 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ae3446-b29d-470b-b4cf-9243fd40874f None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:55.233593 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ae3446-b29d-470b-b4cf-9243fd40874f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:55.234936 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ae3446-b29d-470b-b4cf-9243fd40874f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:55.236144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 285/578] 10.222.0.15 () {56 vars in 1137 bytes} [Tue Sep 13 01:05:55 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:55.236940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642fe74c-c329-4a52-b0ae-1fb16dd473c3 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:55.243609 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 294/579] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:05:55 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:55.244699 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ae3446-b29d-470b-b4cf-9243fd40874f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:55.335938 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642fe74c-c329-4a52-b0ae-1fb16dd473c3 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:55.343739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ae3446-b29d-470b-b4cf-9243fd40874f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:55.362705 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:55.369311 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd47b6b5-5123-4ed2-9f3b-1ddae70caec1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:55.369915 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd47b6b5-5123-4ed2-9f3b-1ddae70caec1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:55.370195 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd47b6b5-5123-4ed2-9f3b-1ddae70caec1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:55.370466 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd47b6b5-5123-4ed2-9f3b-1ddae70caec1 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:55.371173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd47b6b5-5123-4ed2-9f3b-1ddae70caec1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:55.371474 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd47b6b5-5123-4ed2-9f3b-1ddae70caec1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:55.372886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd47b6b5-5123-4ed2-9f3b-1ddae70caec1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:55.373052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:55.378364 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab83068c-9ce5-43b6-969f-c2d3faaef2f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:55.379013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab83068c-9ce5-43b6-969f-c2d3faaef2f7 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:55.379294 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab83068c-9ce5-43b6-969f-c2d3faaef2f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:55.379571 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab83068c-9ce5-43b6-969f-c2d3faaef2f7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:55.384452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 286/580] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:05:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:55.385231 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd47b6b5-5123-4ed2-9f3b-1ddae70caec1 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:55.391360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab83068c-9ce5-43b6-969f-c2d3faaef2f7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:55.392813 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab83068c-9ce5-43b6-969f-c2d3faaef2f7 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:55.408653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-ab83068c-9ce5-43b6-969f-c2d3faaef2f7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:55.408653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:55.408653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:55.408653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:55.408653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:55.408653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:55.408653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:55.408653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:55.408653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:55.408653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:55.408653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:55.408653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:55.408653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:55.408653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:55.408653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:55.408653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:55.409709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:55.409709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:55.409709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:55.409709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:55.409709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:55.409709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:55.409709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:55.409709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:55.409709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:55.409709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:55.409709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:55.409709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:55.409709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:55.409709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:55.409709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:55.409709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:55.409709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:55.410578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:55.410578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:55.410578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:55.410578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 295/581] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:55.410578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab83068c-9ce5-43b6-969f-c2d3faaef2f7 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:55.488583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd47b6b5-5123-4ed2-9f3b-1ddae70caec1 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:55.508900 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab83068c-9ce5-43b6-969f-c2d3faaef2f7 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:55.521807 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcb722a-2bf4-4a93-a1c1-026c198c344e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:55.522434 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcb722a-2bf4-4a93-a1c1-026c198c344e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:55.523262 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcb722a-2bf4-4a93-a1c1-026c198c344e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:55.523548 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcb722a-2bf4-4a93-a1c1-026c198c344e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:55.530006 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bcb722a-2bf4-4a93-a1c1-026c198c344e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:55.531491 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bcb722a-2bf4-4a93-a1c1-026c198c344e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:55.541624 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892faca6-c5ba-409a-bbb5-581f5904a683 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:55.542248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892faca6-c5ba-409a-bbb5-581f5904a683 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:55.542414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-4bcb722a-2bf4-4a93-a1c1-026c198c344e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:05:55.542414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:55.542414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:55.542414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:55.542414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:55.542414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:55.542414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:55.542414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:55.542414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:55.542414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:55.542414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:55.542414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:55.542414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:05:55.542414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:05:55.542414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:05:55.543489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:05:55.543489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:55.543489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:55.543489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:55.543489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:55.543489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:55.543489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:55.543489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:55.543489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:55.543489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:55.543489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:55.543489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:55.543489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:55.543489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:55.543489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:55.543489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:55.543489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:55.544354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:05:55.544354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:05:55.544354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:05:55.544354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:05:55.544354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:05:55.544354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:05:55.544354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:05:55.544354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:55.545117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892faca6-c5ba-409a-bbb5-581f5904a683 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:55.545117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892faca6-c5ba-409a-bbb5-581f5904a683 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:55.548510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 287/582] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:55.548937 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcb722a-2bf4-4a93-a1c1-026c198c344e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:55.553420 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892faca6-c5ba-409a-bbb5-581f5904a683 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:55.555394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892faca6-c5ba-409a-bbb5-581f5904a683 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:55.569579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-892faca6-c5ba-409a-bbb5-581f5904a683 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:55.569579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:55.569579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:55.569579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:55.569579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:55.569579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:55.569579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:55.569579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:55.569579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:55.569579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:55.569579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:55.569579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:55.569579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:55.569579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:55.569579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:55.569579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:55.570497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:55.570497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:55.570497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:55.570497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:55.570497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:55.570497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:55.570497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:55.570497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:55.570497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:55.570497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:55.570497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:55.570497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:55.570497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:55.570497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:55.570497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:55.570497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:55.570497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:55.571400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:55.571400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:55.571400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:55.571400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 296/583] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:55.571626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892faca6-c5ba-409a-bbb5-581f5904a683 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:55.602778 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:55.606562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:55.651747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcb722a-2bf4-4a93-a1c1-026c198c344e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:55.670295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892faca6-c5ba-409a-bbb5-581f5904a683 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:55.684711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1f53b0-01fd-4386-b751-e9f6ba7e0979 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:55.685334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1f53b0-01fd-4386-b751-e9f6ba7e0979 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:55.685617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1f53b0-01fd-4386-b751-e9f6ba7e0979 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:55.685890 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1f53b0-01fd-4386-b751-e9f6ba7e0979 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:55.696993 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1f53b0-01fd-4386-b751-e9f6ba7e0979 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:55.698418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1f53b0-01fd-4386-b751-e9f6ba7e0979 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:55.703436 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a33d14-598d-48b6-90a6-4799b4fd365e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:55.704034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a33d14-598d-48b6-90a6-4799b4fd365e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:55.704311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a33d14-598d-48b6-90a6-4799b4fd365e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:55.704581 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a33d14-598d-48b6-90a6-4799b4fd365e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:55.705295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a33d14-598d-48b6-90a6-4799b4fd365e None admin] RBAC: Adding query filter params (role.id=71b1e7c4b5af44ddbb7a517760ad0119, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=e0d7d8ed4ede4ddca192fce0a497689d) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:55.705598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a33d14-598d-48b6-90a6-4799b4fd365e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:55.707012 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a33d14-598d-48b6-90a6-4799b4fd365e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:55.712298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 297/584] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:05:55 2022] GET /identity/v3/role_assignments?user.id=e0d7d8ed4ede4ddca192fce0a497689d&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=71b1e7c4b5af44ddbb7a517760ad0119 => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:55.712577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-5e1f53b0-01fd-4386-b751-e9f6ba7e0979 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:55.712577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:55.712577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:55.712577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:55.712577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:55.712577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:55.712577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:55.712577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:55.712577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:55.712577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:55.712577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:55.712577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:55.712577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:55.712577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:55.712577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:55.712577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:55.713463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:55.713463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:55.713463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:55.713463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:55.713463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:55.713463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:55.713463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:55.713463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:55.713463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:55.713463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:55.713463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:55.713463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:55.713463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:55.713463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:55.713463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:55.713463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:55.713463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:55.714339 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:55.714339 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:55.714339 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:55.714519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a33d14-598d-48b6-90a6-4799b4fd365e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:55.714589 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 288/585] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:55.714589 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1f53b0-01fd-4386-b751-e9f6ba7e0979 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:55.829314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:55.830700 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1f53b0-01fd-4386-b751-e9f6ba7e0979 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:55.832397 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a33d14-598d-48b6-90a6-4799b4fd365e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:55.832677 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:55.869402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa8fbc9-8ee1-4d18-9ed8-de9188e5ecac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:55.870022 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa8fbc9-8ee1-4d18-9ed8-de9188e5ecac None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:55.870304 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa8fbc9-8ee1-4d18-9ed8-de9188e5ecac None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:55.870577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa8fbc9-8ee1-4d18-9ed8-de9188e5ecac None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:55.871311 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa8fbc9-8ee1-4d18-9ed8-de9188e5ecac None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:55.871634 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa8fbc9-8ee1-4d18-9ed8-de9188e5ecac None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:55.872763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c4acd1-1cb4-4e3f-bd22-47495d07bd25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:55.873057 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa8fbc9-8ee1-4d18-9ed8-de9188e5ecac None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:55.873382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c4acd1-1cb4-4e3f-bd22-47495d07bd25 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:55.873659 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c4acd1-1cb4-4e3f-bd22-47495d07bd25 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:55.873928 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c4acd1-1cb4-4e3f-bd22-47495d07bd25 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:55.874565 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c4acd1-1cb4-4e3f-bd22-47495d07bd25 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:55.874889 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c4acd1-1cb4-4e3f-bd22-47495d07bd25 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:55.876323 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c4acd1-1cb4-4e3f-bd22-47495d07bd25 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:55.894291 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 289/586] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:05:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:55.894291 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa8fbc9-8ee1-4d18-9ed8-de9188e5ecac None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:55.894561 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 298/587] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:55.895448 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c4acd1-1cb4-4e3f-bd22-47495d07bd25 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:55.997646 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa8fbc9-8ee1-4d18-9ed8-de9188e5ecac None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:56.001260 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c4acd1-1cb4-4e3f-bd22-47495d07bd25 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:56.040775 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74f2acd-d46c-4bd1-9a46-d06c324e30a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:56.041400 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74f2acd-d46c-4bd1-9a46-d06c324e30a9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:56.041680 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74f2acd-d46c-4bd1-9a46-d06c324e30a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:56.041956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74f2acd-d46c-4bd1-9a46-d06c324e30a9 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:56.042635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74f2acd-d46c-4bd1-9a46-d06c324e30a9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:56.042972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74f2acd-d46c-4bd1-9a46-d06c324e30a9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:56.044351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74f2acd-d46c-4bd1-9a46-d06c324e30a9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:56.048747 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e690a878-fb25-42e4-8b6b-2ab26a8cd700 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:56.049376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e690a878-fb25-42e4-8b6b-2ab26a8cd700 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:56.049655 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e690a878-fb25-42e4-8b6b-2ab26a8cd700 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:56.049927 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e690a878-fb25-42e4-8b6b-2ab26a8cd700 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:56.050632 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e690a878-fb25-42e4-8b6b-2ab26a8cd700 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:56.050957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e690a878-fb25-42e4-8b6b-2ab26a8cd700 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:56.052367 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e690a878-fb25-42e4-8b6b-2ab26a8cd700 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:56.080754 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 299/588] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:56.080754 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e690a878-fb25-42e4-8b6b-2ab26a8cd700 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:56.080983 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 290/589] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:56.082121 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74f2acd-d46c-4bd1-9a46-d06c324e30a9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:56.105678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:56.117959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:56.202093 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e690a878-fb25-42e4-8b6b-2ab26a8cd700 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:56.232549 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74f2acd-d46c-4bd1-9a46-d06c324e30a9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:56.243517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e0e4f9-1e3b-464a-a78a-c6e56ba509eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:56.244781 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e0e4f9-1e3b-464a-a78a-c6e56ba509eb None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:56.245358 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e0e4f9-1e3b-464a-a78a-c6e56ba509eb None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:56.245815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e0e4f9-1e3b-464a-a78a-c6e56ba509eb None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:56.254813 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e0e4f9-1e3b-464a-a78a-c6e56ba509eb None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:56.256607 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e0e4f9-1e3b-464a-a78a-c6e56ba509eb None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:56.269533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-b4e0e4f9-1e3b-464a-a78a-c6e56ba509eb None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 01:05:56.269533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:56.269533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:56.269533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:56.269533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:56.269533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:56.269533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:56.269533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:56.269533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:56.269533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:56.269533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:56.269533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:56.269533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:05:56.269533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:05:56.269533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:05:56.269533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:05:56.270528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:56.270528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:56.270528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:05:56.270528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:56.270528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:05:56.270528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:56.270528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:56.270528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:56.270528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:56.270528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:56.270528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:56.270528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:56.270528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:56.270528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:56.270528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:56.270528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:56.270528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:56.270528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:56.271528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:56.271528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:56.271528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:05:56.271528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:05:56.271528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:05:56.271528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:05:56.271528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:05:56.271528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:05:56.271528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 01:05:56.271528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:56.273892 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 300/590] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:05:56 2022] GET /identity/v3/users/cinder => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:56.274919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e0e4f9-1e3b-464a-a78a-c6e56ba509eb None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:56.281872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e2aaad-b53f-46c1-91dc-d2594f5f8a4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:56.282543 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e2aaad-b53f-46c1-91dc-d2594f5f8a4e None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:56.282824 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e2aaad-b53f-46c1-91dc-d2594f5f8a4e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:56.283125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e2aaad-b53f-46c1-91dc-d2594f5f8a4e None admin] PATH_INFO: `/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/2f2da4d2d3604d16b58dc25b8c17c49a/roles/71b1e7c4b5af44ddbb7a517760ad0119` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:56.298123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e2aaad-b53f-46c1-91dc-d2594f5f8a4e None admin] RBAC: Authorizing `identity:create_grant(project_id=e5c1519fdae742f4a25eb7cd5476a8cb, user_id=2f2da4d2d3604d16b58dc25b8c17c49a, role_id=71b1e7c4b5af44ddbb7a517760ad0119)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:56.299598 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e2aaad-b53f-46c1-91dc-d2594f5f8a4e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:56.313794 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 291/591] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:05:56 2022] PUT /identity/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/2f2da4d2d3604d16b58dc25b8c17c49a/roles/71b1e7c4b5af44ddbb7a517760ad0119 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:05:56.314583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e2aaad-b53f-46c1-91dc-d2594f5f8a4e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:56.435232 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:56.449585 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:56.482159 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e0e4f9-1e3b-464a-a78a-c6e56ba509eb None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:56.510769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e2aaad-b53f-46c1-91dc-d2594f5f8a4e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:56.536169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054306b7-3ff6-4436-90d8-37107f496f68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:56.536819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054306b7-3ff6-4436-90d8-37107f496f68 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:56.537562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054306b7-3ff6-4436-90d8-37107f496f68 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:56.537841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054306b7-3ff6-4436-90d8-37107f496f68 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:56.547595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054306b7-3ff6-4436-90d8-37107f496f68 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:56.547595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054306b7-3ff6-4436-90d8-37107f496f68 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:56.557295 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7c1ee5-1e33-41d6-839b-77184386dd32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:56.557929 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7c1ee5-1e33-41d6-839b-77184386dd32 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:56.558228 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7c1ee5-1e33-41d6-839b-77184386dd32 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:56.558476 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7c1ee5-1e33-41d6-839b-77184386dd32 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:56.559674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-054306b7-3ff6-4436-90d8-37107f496f68 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:05:56.559674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:56.559674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:56.559674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:56.559674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:56.559674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:56.559674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:56.559674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:56.559674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:56.559674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:56.559674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:56.559674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:56.559674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:05:56.559674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:05:56.559674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:05:56.561267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:05:56.561267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:56.561267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:56.561267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:56.561267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:56.561267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:56.561267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:56.561267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:56.561267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:56.561267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:56.561267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:56.561267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:56.561267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:56.561267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:56.561267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:56.561267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:56.561267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:56.562163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:05:56.562163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:05:56.562163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:05:56.562163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:05:56.562163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:05:56.562163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:05:56.562163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:05:56.562163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:56.562163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 301/592] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:56.562163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054306b7-3ff6-4436-90d8-37107f496f68 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:56.572703 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7c1ee5-1e33-41d6-839b-77184386dd32 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:56.574263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7c1ee5-1e33-41d6-839b-77184386dd32 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:56.588043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-dd7c1ee5-1e33-41d6-839b-77184386dd32 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 13 01:05:56.588043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:56.588043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:56.588043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:56.588043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:56.588043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:56.588043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:56.588043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:56.588043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:56.588043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:56.588043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:56.588043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:56.588043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:05:56.588043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:05:56.588043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:05:56.588043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:05:56.589054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:56.589054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:56.589054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:05:56.589054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:56.589054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:05:56.589054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:56.589054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:56.589054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:56.589054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:56.589054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:56.589054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:56.589054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:56.589054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:56.589054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:56.589054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:56.589054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:56.589054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:56.589054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:56.590005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:56.590005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:56.590005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:05:56.590005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:05:56.590005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:05:56.590005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:05:56.590005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:05:56.590005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:05:56.590005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 13 01:05:56.590005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:56.590005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 292/593] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:05:56 2022] GET /identity/v3/users/neutron => generated 85 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:56.590837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7c1ee5-1e33-41d6-839b-77184386dd32 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:56.690623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054306b7-3ff6-4436-90d8-37107f496f68 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:56.707537 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7c1ee5-1e33-41d6-839b-77184386dd32 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:56.725360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3b84be-a34e-4588-93a4-b59a8a7d45e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:56.726005 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3b84be-a34e-4588-93a4-b59a8a7d45e8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:56.726285 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3b84be-a34e-4588-93a4-b59a8a7d45e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:56.726559 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3b84be-a34e-4588-93a4-b59a8a7d45e8 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:56.727317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3b84be-a34e-4588-93a4-b59a8a7d45e8 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:56.727625 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3b84be-a34e-4588-93a4-b59a8a7d45e8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:56.729042 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3b84be-a34e-4588-93a4-b59a8a7d45e8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:56.738010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:56.744312 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:56.745782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9ec9a3-bdae-4fe2-b415-0e2dd22de1ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:56.745782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9ec9a3-bdae-4fe2-b415-0e2dd22de1ee None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:56.746068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9ec9a3-bdae-4fe2-b415-0e2dd22de1ee None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:56.746516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9ec9a3-bdae-4fe2-b415-0e2dd22de1ee None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:56.747379 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9ec9a3-bdae-4fe2-b415-0e2dd22de1ee None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:56.747755 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9ec9a3-bdae-4fe2-b415-0e2dd22de1ee None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:56.748335 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 302/594] 10.222.0.15 () {56 vars in 1137 bytes} [Tue Sep 13 01:05:56 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:56.749140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3b84be-a34e-4588-93a4-b59a8a7d45e8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:56.749404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9ec9a3-bdae-4fe2-b415-0e2dd22de1ee None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:56.765353 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 293/595] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:05:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:56.770902 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9ec9a3-bdae-4fe2-b415-0e2dd22de1ee None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:56.852695 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3b84be-a34e-4588-93a4-b59a8a7d45e8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:56.881339 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9ec9a3-bdae-4fe2-b415-0e2dd22de1ee None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:56.891337 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba826799-642c-40ad-800a-c17b4405ad97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:56.892188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba826799-642c-40ad-800a-c17b4405ad97 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:56.892665 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba826799-642c-40ad-800a-c17b4405ad97 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:56.893220 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba826799-642c-40ad-800a-c17b4405ad97 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:56.894193 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba826799-642c-40ad-800a-c17b4405ad97 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:56.894697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba826799-642c-40ad-800a-c17b4405ad97 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:56.896595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba826799-642c-40ad-800a-c17b4405ad97 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:56.916132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 303/596] 10.222.0.15 () {56 vars in 1139 bytes} [Tue Sep 13 01:05:56 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:56.917245 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba826799-642c-40ad-800a-c17b4405ad97 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:56.922990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a265c1b-16e5-4060-bc98-3e4710632553 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:56.923632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a265c1b-16e5-4060-bc98-3e4710632553 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:56.923914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a265c1b-16e5-4060-bc98-3e4710632553 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:56.924187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a265c1b-16e5-4060-bc98-3e4710632553 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:56.938599 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a265c1b-16e5-4060-bc98-3e4710632553 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:56.938599 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a265c1b-16e5-4060-bc98-3e4710632553 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:56.956837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-3a265c1b-16e5-4060-bc98-3e4710632553 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:56.956837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:56.956837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:56.956837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:56.956837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:56.956837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:56.956837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:56.956837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:56.956837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:56.956837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:56.956837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:56.956837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:56.956837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:56.956837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:56.956837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:56.956837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:56.957793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:56.957793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:56.957793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:56.957793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:56.957793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:56.957793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:56.957793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:56.957793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:56.957793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:56.957793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:56.957793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:56.957793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:56.957793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:56.957793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:56.957793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:56.957793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:56.957793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:56.958683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:56.958683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:56.958683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:56.958683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 294/597] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:56.960003 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a265c1b-16e5-4060-bc98-3e4710632553 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:56.986595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:57.010214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:57.041993 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba826799-642c-40ad-800a-c17b4405ad97 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:57.073406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a265c1b-16e5-4060-bc98-3e4710632553 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:57.080603 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6d8879-cb85-4f37-8376-e25ef9284d73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:57.081372 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6d8879-cb85-4f37-8376-e25ef9284d73 None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:57.081658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6d8879-cb85-4f37-8376-e25ef9284d73 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:57.081934 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6d8879-cb85-4f37-8376-e25ef9284d73 None admin] PATH_INFO: `/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/eef98bcf2699461d9a7f3c3bffe82f34/roles/71b1e7c4b5af44ddbb7a517760ad0119` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:57.097911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6d8879-cb85-4f37-8376-e25ef9284d73 None admin] RBAC: Authorizing `identity:create_grant(project_id=e5c1519fdae742f4a25eb7cd5476a8cb, user_id=eef98bcf2699461d9a7f3c3bffe82f34, role_id=71b1e7c4b5af44ddbb7a517760ad0119)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:57.099557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6d8879-cb85-4f37-8376-e25ef9284d73 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:57.110241 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336cebc0-65d6-49a5-9f21-323e81691436 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:57.111020 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336cebc0-65d6-49a5-9f21-323e81691436 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:57.111302 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336cebc0-65d6-49a5-9f21-323e81691436 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:57.111572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336cebc0-65d6-49a5-9f21-323e81691436 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:57.115411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 304/598] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:05:56 2022] PUT /identity/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/eef98bcf2699461d9a7f3c3bffe82f34/roles/71b1e7c4b5af44ddbb7a517760ad0119 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:05:57.116299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6d8879-cb85-4f37-8376-e25ef9284d73 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:57.142343 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336cebc0-65d6-49a5-9f21-323e81691436 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:57.143953 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336cebc0-65d6-49a5-9f21-323e81691436 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:57.180175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-336cebc0-65d6-49a5-9f21-323e81691436 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:57.180175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:57.180175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:57.180175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:57.180175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:57.180175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:57.180175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:57.180175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:57.180175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:57.180175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:57.180175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:57.180175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:57.180175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:57.180175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:57.180175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:57.180175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:57.182713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:57.182713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:57.182713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:57.182713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:57.182713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:57.182713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:57.182713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:57.182713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:57.182713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:57.182713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:57.182713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:57.182713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:57.182713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:57.182713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:57.182713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:57.182713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:57.182713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:57.184866 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:57.184866 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:57.184866 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:57.184866 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 295/599] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:57.188286 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336cebc0-65d6-49a5-9f21-323e81691436 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:57.340186 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6d8879-cb85-4f37-8376-e25ef9284d73 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:57.347951 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:57.366373 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:57.373533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336cebc0-65d6-49a5-9f21-323e81691436 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:57.378144 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6b6dc2-5914-4eaf-946b-0b6f48750c18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:57.378769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6b6dc2-5914-4eaf-946b-0b6f48750c18 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:57.379084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6b6dc2-5914-4eaf-946b-0b6f48750c18 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:57.379361 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6b6dc2-5914-4eaf-946b-0b6f48750c18 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:57.380179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6b6dc2-5914-4eaf-946b-0b6f48750c18 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:57.380490 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6b6dc2-5914-4eaf-946b-0b6f48750c18 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:57.381861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6b6dc2-5914-4eaf-946b-0b6f48750c18 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:57.394456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 305/600] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:57.399934 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6b6dc2-5914-4eaf-946b-0b6f48750c18 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:57.418762 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8219d499-dcc7-4d8c-8f5c-c7f2ce65e8be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:57.419404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8219d499-dcc7-4d8c-8f5c-c7f2ce65e8be None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:57.419683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8219d499-dcc7-4d8c-8f5c-c7f2ce65e8be None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:57.419954 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8219d499-dcc7-4d8c-8f5c-c7f2ce65e8be None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:57.420808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8219d499-dcc7-4d8c-8f5c-c7f2ce65e8be None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:57.421112 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8219d499-dcc7-4d8c-8f5c-c7f2ce65e8be None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:57.422628 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8219d499-dcc7-4d8c-8f5c-c7f2ce65e8be None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:57.436245 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 296/601] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:57.442034 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8219d499-dcc7-4d8c-8f5c-c7f2ce65e8be None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:57.505805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6b6dc2-5914-4eaf-946b-0b6f48750c18 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:57.541224 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605eb1b1-4e5e-4710-a302-4a7181d06e28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:57.541893 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605eb1b1-4e5e-4710-a302-4a7181d06e28 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:57.542176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605eb1b1-4e5e-4710-a302-4a7181d06e28 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:57.542451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605eb1b1-4e5e-4710-a302-4a7181d06e28 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:57.549496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605eb1b1-4e5e-4710-a302-4a7181d06e28 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:57.550951 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605eb1b1-4e5e-4710-a302-4a7181d06e28 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:57.553789 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8219d499-dcc7-4d8c-8f5c-c7f2ce65e8be None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:57.563382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-605eb1b1-4e5e-4710-a302-4a7181d06e28 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:05:57.563382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:57.563382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:57.563382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:57.563382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:57.563382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:57.563382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:57.563382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:57.563382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:57.563382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:57.563382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:57.563382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:57.563382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:05:57.563382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:05:57.563382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:05:57.565399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:05:57.565399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:57.565399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:57.565399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:57.565399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:57.565399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:57.565399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:57.565399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:57.565399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:57.565399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:57.565399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:57.565399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:57.565399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:57.565399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:57.565399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:57.565399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:57.565399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:57.567547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:05:57.567547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:05:57.567547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:05:57.567547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:05:57.567547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:05:57.567547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:05:57.567547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:05:57.567547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:57.567547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 306/602] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:05:57.574889 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605eb1b1-4e5e-4710-a302-4a7181d06e28 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:57.592413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a83e42-e072-46c9-a0e5-73f8594eb63d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:57.592965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a83e42-e072-46c9-a0e5-73f8594eb63d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:57.593293 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a83e42-e072-46c9-a0e5-73f8594eb63d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:57.593669 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a83e42-e072-46c9-a0e5-73f8594eb63d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:57.599445 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:57.601981 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a83e42-e072-46c9-a0e5-73f8594eb63d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:57.603681 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a83e42-e072-46c9-a0e5-73f8594eb63d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:57.617206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-27a83e42-e072-46c9-a0e5-73f8594eb63d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:05:57.617206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:57.617206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:57.617206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:57.617206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:57.617206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:57.617206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:57.617206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:57.617206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:57.617206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:57.617206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:57.617206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:57.617206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:05:57.617206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:05:57.617206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:05:57.618130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:05:57.618130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:57.618130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:57.618130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:57.618130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:57.618130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:57.618130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:57.618130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:57.618130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:57.618130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:57.618130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:57.618130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:57.618130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:57.618130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:57.618130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:57.618130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:57.618130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:57.619028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:05:57.619028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:05:57.619028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:05:57.619028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:05:57.619028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:05:57.619028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:05:57.619028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:05:57.619028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:57.621438 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 297/603] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:05:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:05:57.627125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a83e42-e072-46c9-a0e5-73f8594eb63d None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:57.646597 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:57.714664 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605eb1b1-4e5e-4710-a302-4a7181d06e28 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:57.735386 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a83e42-e072-46c9-a0e5-73f8594eb63d None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:57.770948 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550301b1-3427-4f29-93b2-95e66265ac0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:57.772018 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550301b1-3427-4f29-93b2-95e66265ac0b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:57.772481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550301b1-3427-4f29-93b2-95e66265ac0b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:57.773491 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550301b1-3427-4f29-93b2-95e66265ac0b None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:57.774353 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550301b1-3427-4f29-93b2-95e66265ac0b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:57.774837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550301b1-3427-4f29-93b2-95e66265ac0b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:57.776601 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550301b1-3427-4f29-93b2-95e66265ac0b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:57.787570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f96752-3a49-4611-b099-126f411739c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:57.788253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f96752-3a49-4611-b099-126f411739c1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:57.788573 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f96752-3a49-4611-b099-126f411739c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:57.788814 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f96752-3a49-4611-b099-126f411739c1 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:57.789511 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f96752-3a49-4611-b099-126f411739c1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:57.789820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f96752-3a49-4611-b099-126f411739c1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:57.791406 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f96752-3a49-4611-b099-126f411739c1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:57.791603 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 298/604] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:05:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:57.798213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550301b1-3427-4f29-93b2-95e66265ac0b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:57.824928 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 307/605] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:05:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:57.835998 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f96752-3a49-4611-b099-126f411739c1 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:57.891305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:57.914212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550301b1-3427-4f29-93b2-95e66265ac0b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:57.953023 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120198c4-1e3c-4852-8517-bc81d9e494d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:57.954016 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120198c4-1e3c-4852-8517-bc81d9e494d9 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:57.954544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120198c4-1e3c-4852-8517-bc81d9e494d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:57.955056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120198c4-1e3c-4852-8517-bc81d9e494d9 None admin] PATH_INFO: `/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/09cd263f72484733adaaa0850ebfa609/roles/71b1e7c4b5af44ddbb7a517760ad0119` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:57.979732 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120198c4-1e3c-4852-8517-bc81d9e494d9 None admin] RBAC: Authorizing `identity:create_grant(project_id=e5c1519fdae742f4a25eb7cd5476a8cb, user_id=09cd263f72484733adaaa0850ebfa609, role_id=71b1e7c4b5af44ddbb7a517760ad0119)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:57.981866 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120198c4-1e3c-4852-8517-bc81d9e494d9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:58.001630 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 299/606] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:05:57 2022] PUT /identity/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/09cd263f72484733adaaa0850ebfa609/roles/71b1e7c4b5af44ddbb7a517760ad0119 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:05:58.010921 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eac9b3-6b46-4c77-be47-d5b94732243a None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:58.010921 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eac9b3-6b46-4c77-be47-d5b94732243a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:58.010921 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eac9b3-6b46-4c77-be47-d5b94732243a None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:58.011282 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 300/607] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:58 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:58.014947 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc027e2-8191-4915-a5c3-9b9f6c9ce6c4 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:58.015281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc027e2-8191-4915-a5c3-9b9f6c9ce6c4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:58.015556 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc027e2-8191-4915-a5c3-9b9f6c9ce6c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:58.043558 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-bcc027e2-8191-4915-a5c3-9b9f6c9ce6c4 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:58.045471 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-bcc027e2-8191-4915-a5c3-9b9f6c9ce6c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:58.051508 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:58.086119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f96752-3a49-4611-b099-126f411739c1 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:58.168151 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 301/608] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 01:05:58.169152 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6052a3-e38f-460d-a913-051d116f674e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:58.170095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6052a3-e38f-460d-a913-051d116f674e None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:58.170620 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6052a3-e38f-460d-a913-051d116f674e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:58.171119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6052a3-e38f-460d-a913-051d116f674e None admin] PATH_INFO: `/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/f90a5e57daa643eab94c01e21fcbc4c4/roles/71b1e7c4b5af44ddbb7a517760ad0119` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:58.187829 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6052a3-e38f-460d-a913-051d116f674e None admin] RBAC: Authorizing `identity:create_grant(project_id=e5c1519fdae742f4a25eb7cd5476a8cb, user_id=f90a5e57daa643eab94c01e21fcbc4c4, role_id=71b1e7c4b5af44ddbb7a517760ad0119)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:58.189924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6052a3-e38f-460d-a913-051d116f674e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:58.212752 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 308/609] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:05:57 2022] PUT /identity/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/f90a5e57daa643eab94c01e21fcbc4c4/roles/71b1e7c4b5af44ddbb7a517760ad0119 => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:05:58.512534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc027e2-8191-4915-a5c3-9b9f6c9ce6c4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:58.572417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:58.620784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc027e2-8191-4915-a5c3-9b9f6c9ce6c4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:58.662740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82fca2cd-7930-4ce8-a1a0-d56d53b5e9dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:58.663971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fca2cd-7930-4ce8-a1a0-d56d53b5e9dd None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:58.664252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fca2cd-7930-4ce8-a1a0-d56d53b5e9dd None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:58.664520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fca2cd-7930-4ce8-a1a0-d56d53b5e9dd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:58.679908 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82fca2cd-7930-4ce8-a1a0-d56d53b5e9dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:58.685011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-82fca2cd-7930-4ce8-a1a0-d56d53b5e9dd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:58.685011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:58.685011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:58.685011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:58.685011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:58.685011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:58.685011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:58.685011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:58.685011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:58.685011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:58.685011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:58.685011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:58.685011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:58.685011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:58.685011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:58.685011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:58.685950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:58.685950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:58.685950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:58.685950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:58.685950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:58.685950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:58.685950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:58.685950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:58.685950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:58.685950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:58.685950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:58.685950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:58.685950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:58.685950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:58.685950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:58.685950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:58.686795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:58.686795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:58.686795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:58.686795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:58.686795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:58.686795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:58.686795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:58.686795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:58.686795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:58.686795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:58.686795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:58.686795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:58.686795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 302/610] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:05:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:58.692234 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6052a3-e38f-460d-a913-051d116f674e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:58.820882 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6052a3-e38f-460d-a913-051d116f674e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:58.841062 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964c9465-3b90-43e1-98e9-54f3dc1caec3 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:58.841592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964c9465-3b90-43e1-98e9-54f3dc1caec3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:58.841592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964c9465-3b90-43e1-98e9-54f3dc1caec3 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:58.846839 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 303/611] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:58.848083 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69834385-cf91-49aa-bd6d-2ffd812f7fb5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:58.848387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69834385-cf91-49aa-bd6d-2ffd812f7fb5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:58.848673 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69834385-cf91-49aa-bd6d-2ffd812f7fb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:58.862300 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787929f2-72a8-4ad5-a9d1-e8c7bda3e6d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:58.863200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787929f2-72a8-4ad5-a9d1-e8c7bda3e6d9 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:58.863642 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787929f2-72a8-4ad5-a9d1-e8c7bda3e6d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:58.864078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787929f2-72a8-4ad5-a9d1-e8c7bda3e6d9 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:58.864886 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787929f2-72a8-4ad5-a9d1-e8c7bda3e6d9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:58.865346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787929f2-72a8-4ad5-a9d1-e8c7bda3e6d9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:58.867525 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787929f2-72a8-4ad5-a9d1-e8c7bda3e6d9 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:58.870202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:58.877246 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 309/612] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:05:58 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:58.882657 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787929f2-72a8-4ad5-a9d1-e8c7bda3e6d9 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:58.887970 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-69834385-cf91-49aa-bd6d-2ffd812f7fb5 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:05:58.889172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-69834385-cf91-49aa-bd6d-2ffd812f7fb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:05:58.989655 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 304/613] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:05:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:05:59.073704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787929f2-72a8-4ad5-a9d1-e8c7bda3e6d9 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:59.118162 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1fc7e9-ac88-4976-b19d-1af69a1c1047 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:59.118794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1fc7e9-ac88-4976-b19d-1af69a1c1047 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:59.119098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1fc7e9-ac88-4976-b19d-1af69a1c1047 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:59.119369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1fc7e9-ac88-4976-b19d-1af69a1c1047 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:59.133940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1fc7e9-ac88-4976-b19d-1af69a1c1047 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:59.135472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1fc7e9-ac88-4976-b19d-1af69a1c1047 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:59.156094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-1d1fc7e9-ac88-4976-b19d-1af69a1c1047 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:59.156094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:59.156094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:59.156094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:59.156094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:59.156094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:59.156094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:59.156094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:59.156094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:59.156094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:59.156094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:59.156094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:59.156094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:05:59.156094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:05:59.156094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:05:59.156094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:05:59.157301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:59.157301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:59.157301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:59.157301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:59.157301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:59.157301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:59.157301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:59.157301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:59.157301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:59.157301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:59.157301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:59.157301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:59.157301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:59.157301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:59.157301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:59.157301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:59.157301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:05:59.158406 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:05:59.158406 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:05:59.158406 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:59.158406 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 310/614] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:05:59.163204 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69834385-cf91-49aa-bd6d-2ffd812f7fb5 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:59.239286 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:59.315812 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69834385-cf91-49aa-bd6d-2ffd812f7fb5 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:59.370211 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0582259c-1ee3-4c25-b00f-1b88f16f6212 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:59.371212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0582259c-1ee3-4c25-b00f-1b88f16f6212 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:59.371725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0582259c-1ee3-4c25-b00f-1b88f16f6212 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:59.372014 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0582259c-1ee3-4c25-b00f-1b88f16f6212 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:59.372659 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0582259c-1ee3-4c25-b00f-1b88f16f6212 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:59.372962 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0582259c-1ee3-4c25-b00f-1b88f16f6212 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:59.374454 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0582259c-1ee3-4c25-b00f-1b88f16f6212 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:59.396190 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 305/615] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:05:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:59.402481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1fc7e9-ac88-4976-b19d-1af69a1c1047 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:59.438778 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0582259c-1ee3-4c25-b00f-1b88f16f6212 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:59.550945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:59.553103 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0582259c-1ee3-4c25-b00f-1b88f16f6212 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:59.561261 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:59.604437 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283820d0-04ae-4e79-821d-c65b590476e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:59.605368 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283820d0-04ae-4e79-821d-c65b590476e6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:59.605830 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283820d0-04ae-4e79-821d-c65b590476e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:59.606308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283820d0-04ae-4e79-821d-c65b590476e6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:59.615643 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1fc7e9-ac88-4976-b19d-1af69a1c1047 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:59.628394 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283820d0-04ae-4e79-821d-c65b590476e6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:05:59.642874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-283820d0-04ae-4e79-821d-c65b590476e6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:59.642874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:59.642874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:59.642874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:59.642874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:59.642874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:59.642874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:59.642874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:59.642874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:59.642874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:59.642874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:59.642874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:59.642874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:05:59.642874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:05:59.642874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:05:59.642874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:05:59.643825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:05:59.643825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:05:59.643825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:05:59.643825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:05:59.643825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:59.643825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:59.643825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:59.643825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:59.643825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:59.643825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:59.643825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:59.643825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:59.643825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:59.643825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:59.643825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:59.643825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:59.644645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:59.644645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:59.644645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:59.644645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:59.644645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:05:59.644645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:05:59.644645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:05:59.644645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:05:59.644645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:05:59.644645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:05:59.644645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:05:59.644645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:05:59.647352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 306/616] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:05:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:05:59.651047 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7972d6-9fd0-450d-9880-aaff8adb57d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:59.651907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7972d6-9fd0-450d-9880-aaff8adb57d8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:59.652378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7972d6-9fd0-450d-9880-aaff8adb57d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:59.652831 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7972d6-9fd0-450d-9880-aaff8adb57d8 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:59.653053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283820d0-04ae-4e79-821d-c65b590476e6 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:59.661411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7972d6-9fd0-450d-9880-aaff8adb57d8 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:59.663118 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7972d6-9fd0-450d-9880-aaff8adb57d8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:59.675916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-6a7972d6-9fd0-450d-9880-aaff8adb57d8 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 13 01:05:59.675916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:05:59.675916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:05:59.675916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:05:59.675916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:05:59.675916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:05:59.675916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:05:59.675916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:05:59.675916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:05:59.675916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:05:59.675916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:05:59.675916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:05:59.675916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:05:59.675916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:05:59.675916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:05:59.675916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:05:59.677662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:05:59.677662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:05:59.677662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:05:59.677662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:59.677662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:05:59.677662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:05:59.677662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:05:59.677662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:05:59.677662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:05:59.677662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:05:59.677662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:05:59.677662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:05:59.677662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:05:59.677662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:05:59.677662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:05:59.677662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:05:59.677662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:05:59.677662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:05:59.679288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:05:59.679288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:05:59.679288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:05:59.679288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:05:59.679288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:05:59.679288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:05:59.679288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:05:59.679288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:05:59.679288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 13 01:05:59.679288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:05:59.681922 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 311/617] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:05:59 2022] GET /identity/v3/users/swift => generated 83 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:05:59.692581 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7972d6-9fd0-450d-9880-aaff8adb57d8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:59.792651 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283820d0-04ae-4e79-821d-c65b590476e6 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:59.842900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d897ff0-36ce-478d-b2fc-b78069b84897 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:59.842900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d897ff0-36ce-478d-b2fc-b78069b84897 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:59.842900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d897ff0-36ce-478d-b2fc-b78069b84897 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:59.842900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d897ff0-36ce-478d-b2fc-b78069b84897 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:59.843818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d897ff0-36ce-478d-b2fc-b78069b84897 None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:59.843818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d897ff0-36ce-478d-b2fc-b78069b84897 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:59.843818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d897ff0-36ce-478d-b2fc-b78069b84897 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:59.852249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 307/618] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:05:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:05:59.857754 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d897ff0-36ce-478d-b2fc-b78069b84897 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:05:59.892371 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:59.908987 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7972d6-9fd0-450d-9880-aaff8adb57d8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:59.920631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:05:59.947626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab989813-e994-43c5-bf7a-7a216ea64993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:05:59.948286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab989813-e994-43c5-bf7a-7a216ea64993 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:59.948667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab989813-e994-43c5-bf7a-7a216ea64993 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:59.948940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab989813-e994-43c5-bf7a-7a216ea64993 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:59.949852 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab989813-e994-43c5-bf7a-7a216ea64993 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:05:59.950156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab989813-e994-43c5-bf7a-7a216ea64993 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:05:59.951613 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab989813-e994-43c5-bf7a-7a216ea64993 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:05:59.972898 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d897ff0-36ce-478d-b2fc-b78069b84897 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:05:59.973785 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 312/619] 10.222.0.15 () {56 vars in 1135 bytes} [Tue Sep 13 01:05:59 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:05:59.976680 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1a7a5e-6f62-4f7b-be8c-46a7f4c4adc6 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:05:59.976992 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1a7a5e-6f62-4f7b-be8c-46a7f4c4adc6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:05:59.977255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1a7a5e-6f62-4f7b-be8c-46a7f4c4adc6 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:05:59.982939 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 313/620] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:05:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:05:59.985389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1a7a5e-6f62-4f7b-be8c-46a7f4c4adc6 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:00.016798 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebbde403-6bfe-4021-b302-0b71609dc73d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:00.017831 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbde403-6bfe-4021-b302-0b71609dc73d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:00.018357 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbde403-6bfe-4021-b302-0b71609dc73d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:00.018638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbde403-6bfe-4021-b302-0b71609dc73d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:00.035456 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebbde403-6bfe-4021-b302-0b71609dc73d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:00.036986 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebbde403-6bfe-4021-b302-0b71609dc73d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:00.057963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-ebbde403-6bfe-4021-b302-0b71609dc73d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:00.057963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:00.057963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:00.057963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:00.057963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:00.057963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:00.057963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:00.057963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:00.057963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:00.057963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:00.057963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:00.057963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:00.057963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:00.057963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:00.057963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:00.057963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:00.059082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:00.059082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:00.059082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:00.059082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:00.059082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:00.059082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:00.059082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:00.059082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:00.059082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:00.059082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:00.059082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:00.059082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:00.059082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:00.059082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:00.059082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:00.059082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:00.059082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:00.059960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:00.059960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:00.059960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:00.060129 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 308/621] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:00.061634 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c12fa4-3b62-4363-9fbf-2e83db534a27 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:00.062260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c12fa4-3b62-4363-9fbf-2e83db534a27 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:00.062532 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c12fa4-3b62-4363-9fbf-2e83db534a27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:00.085552 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-60c12fa4-3b62-4363-9fbf-2e83db534a27 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:00.087172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-60c12fa4-3b62-4363-9fbf-2e83db534a27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:00.118245 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1a7a5e-6f62-4f7b-be8c-46a7f4c4adc6 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:00.163177 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0603ca46-8dd6-413d-94d8-f8a5e89269b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:00.163732 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0603ca46-8dd6-413d-94d8-f8a5e89269b4 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:00.164007 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0603ca46-8dd6-413d-94d8-f8a5e89269b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:00.164275 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0603ca46-8dd6-413d-94d8-f8a5e89269b4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:00.174660 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0603ca46-8dd6-413d-94d8-f8a5e89269b4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:00.175118 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 309/622] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:06:00.176147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c12fa4-3b62-4363-9fbf-2e83db534a27 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:00.176335 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0603ca46-8dd6-413d-94d8-f8a5e89269b4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:00.194282 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:00.197291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-0603ca46-8dd6-413d-94d8-f8a5e89269b4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:00.197291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:00.197291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:00.197291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:00.197291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:00.197291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:00.197291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:00.197291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:00.197291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:00.197291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:00.197291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:00.197291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:00.197291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:00.197291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:00.197291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:00.197291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:00.199229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:00.199229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:00.199229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:00.199229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:00.199229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:00.199229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:00.199229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:00.199229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:00.199229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:00.199229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:00.199229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:00.199229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:00.199229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:00.199229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:00.199229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:00.199229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:00.199229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:00.200381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:00.200381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:00.200381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:00.200381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 314/623] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:05:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:00.203819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0603ca46-8dd6-413d-94d8-f8a5e89269b4 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:00.228115 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:00.310652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0603ca46-8dd6-413d-94d8-f8a5e89269b4 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:00.348017 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b2fc6c-c6a3-4449-ab3c-203d5acffd6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:00.348793 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b2fc6c-c6a3-4449-ab3c-203d5acffd6a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:00.349167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b2fc6c-c6a3-4449-ab3c-203d5acffd6a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:00.349530 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b2fc6c-c6a3-4449-ab3c-203d5acffd6a None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:00.350284 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b2fc6c-c6a3-4449-ab3c-203d5acffd6a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:00.350710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b2fc6c-c6a3-4449-ab3c-203d5acffd6a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:00.352195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b2fc6c-c6a3-4449-ab3c-203d5acffd6a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:00.365345 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 315/624] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:00.370298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b2fc6c-c6a3-4449-ab3c-203d5acffd6a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:00.412818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c12fa4-3b62-4363-9fbf-2e83db534a27 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:00.467308 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:00.488446 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba85439b-691d-4e0e-89ca-103d5ef98a62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:00.489168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba85439b-691d-4e0e-89ca-103d5ef98a62 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:00.489515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba85439b-691d-4e0e-89ca-103d5ef98a62 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:00.489852 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba85439b-691d-4e0e-89ca-103d5ef98a62 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:00.490495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba85439b-691d-4e0e-89ca-103d5ef98a62 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:00.490791 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba85439b-691d-4e0e-89ca-103d5ef98a62 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:00.492706 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba85439b-691d-4e0e-89ca-103d5ef98a62 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:00.500553 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b2fc6c-c6a3-4449-ab3c-203d5acffd6a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:00.526016 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 310/625] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 350 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:00.529076 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba85439b-691d-4e0e-89ca-103d5ef98a62 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:00.562233 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c278c7bf-b5eb-4d79-80e5-079dbf9b5ed3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:00.562897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c278c7bf-b5eb-4d79-80e5-079dbf9b5ed3 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:00.564003 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c278c7bf-b5eb-4d79-80e5-079dbf9b5ed3 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:00.564283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c278c7bf-b5eb-4d79-80e5-079dbf9b5ed3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:00.572688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c278c7bf-b5eb-4d79-80e5-079dbf9b5ed3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:00.574931 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c278c7bf-b5eb-4d79-80e5-079dbf9b5ed3 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:00.589605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-c278c7bf-b5eb-4d79-80e5-079dbf9b5ed3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:00.589605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:00.589605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:00.589605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:00.589605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:00.589605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:00.589605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:00.589605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:00.589605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:00.589605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:00.589605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:00.589605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:00.589605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:00.589605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:00.589605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:00.590533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:00.590533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:00.590533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:00.590533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:00.590533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:00.590533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:00.590533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:00.590533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:00.590533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:00.590533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:00.590533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:00.590533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:00.590533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:00.590533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:00.590533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:00.590533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:00.590533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:00.591446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:00.591446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:00.591446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:00.591446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:00.591446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:00.591446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:00.591446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:00.591446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:00.591876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 316/626] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:00.592453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c278c7bf-b5eb-4d79-80e5-079dbf9b5ed3 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:00.624914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:00.659379 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba85439b-691d-4e0e-89ca-103d5ef98a62 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:00.705652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3c5bf9-86b9-4443-9ceb-0d85077c4410 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:00.706660 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3c5bf9-86b9-4443-9ceb-0d85077c4410 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:00.706963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3c5bf9-86b9-4443-9ceb-0d85077c4410 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:00.707235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3c5bf9-86b9-4443-9ceb-0d85077c4410 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:00.718153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3c5bf9-86b9-4443-9ceb-0d85077c4410 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:00.720255 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3c5bf9-86b9-4443-9ceb-0d85077c4410 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:00.721398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c278c7bf-b5eb-4d79-80e5-079dbf9b5ed3 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:00.734632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-3a3c5bf9-86b9-4443-9ceb-0d85077c4410 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 01:06:00.734632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:00.734632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:00.734632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:00.734632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:00.734632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:00.734632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:00.734632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:00.734632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:00.734632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:00.734632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:00.734632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:00.734632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:00.734632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:00.734632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:00.734632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:00.735639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:00.735639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:00.735639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:00.735639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:00.735639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:00.735639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:00.735639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:00.735639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:00.735639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:00.735639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:00.735639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:00.735639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:00.735639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:00.735639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:00.735639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:00.735639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:00.735639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:00.735639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:00.736565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:00.736565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:00.736565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:00.736565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:00.736565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:00.736565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:00.736565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:00.736565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:00.736565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 01:06:00.736565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:00.736565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 311/627] 10.222.0.15 () {56 vars in 1093 bytes} [Tue Sep 13 01:06:00 2022] GET /identity/v3/users/nova => generated 82 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:00.737712 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3c5bf9-86b9-4443-9ceb-0d85077c4410 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:00.759796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ad4d52-8139-4711-a893-db2b810d0a9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:00.760705 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ad4d52-8139-4711-a893-db2b810d0a9e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:00.760991 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ad4d52-8139-4711-a893-db2b810d0a9e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:00.761259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ad4d52-8139-4711-a893-db2b810d0a9e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:00.768949 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:00.774314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ad4d52-8139-4711-a893-db2b810d0a9e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:00.779502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-09ad4d52-8139-4711-a893-db2b810d0a9e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:06:00.779502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:00.779502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:00.779502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:00.779502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:00.779502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:00.779502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:00.779502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:00.779502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:00.779502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:00.779502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:00.779502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:00.779502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:00.779502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:00.779502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:00.779502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:00.780404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:00.780404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:00.780404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:00.780404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:00.780404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:00.780404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:00.780404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:00.780404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:00.780404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:00.780404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:00.780404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:00.780404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:00.780404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:00.780404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:00.780404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:00.780404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:00.781223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:00.781223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:00.781223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:00.781223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:00.781223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:00.781223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:00.781223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:00.781223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:00.781223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:00.781223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:00.781223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:06:00.781223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:00.785474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 317/628] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:06:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:00.786317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ad4d52-8139-4711-a893-db2b810d0a9e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:00.852654 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3c5bf9-86b9-4443-9ceb-0d85077c4410 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:00.879887 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:00.886524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54aa190-0622-4499-808b-2501d822ab4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:00.887167 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54aa190-0622-4499-808b-2501d822ab4b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:00.887446 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54aa190-0622-4499-808b-2501d822ab4b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:00.887715 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54aa190-0622-4499-808b-2501d822ab4b None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:00.888402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54aa190-0622-4499-808b-2501d822ab4b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:00.888709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54aa190-0622-4499-808b-2501d822ab4b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:00.890135 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54aa190-0622-4499-808b-2501d822ab4b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:00.900569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ad4d52-8139-4711-a893-db2b810d0a9e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:00.902222 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 312/629] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:00.908213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ad760f-3487-4696-9f7f-9221be5f1ed8 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:00.908495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ad760f-3487-4696-9f7f-9221be5f1ed8 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:00.908765 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ad760f-3487-4696-9f7f-9221be5f1ed8 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:00.910006 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 313/630] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:00 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:00.910692 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ad760f-3487-4696-9f7f-9221be5f1ed8 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:00.941498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f055b7-7537-45c5-821a-3bf774c34bad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:00.942105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f055b7-7537-45c5-821a-3bf774c34bad None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:00.942379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f055b7-7537-45c5-821a-3bf774c34bad None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:00.942650 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f055b7-7537-45c5-821a-3bf774c34bad None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:00.943408 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f055b7-7537-45c5-821a-3bf774c34bad None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:00.943712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f055b7-7537-45c5-821a-3bf774c34bad None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:00.945169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f055b7-7537-45c5-821a-3bf774c34bad None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:00.962418 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 318/631] 10.222.0.15 () {56 vars in 1133 bytes} [Tue Sep 13 01:06:00 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:00.963213 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f055b7-7537-45c5-821a-3bf774c34bad None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:01.011396 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ad760f-3487-4696-9f7f-9221be5f1ed8 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:01.022156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:01.046173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5705c2-4802-449b-978f-75b436f2be7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:01.046776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5705c2-4802-449b-978f-75b436f2be7b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:01.047086 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5705c2-4802-449b-978f-75b436f2be7b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:01.047355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5705c2-4802-449b-978f-75b436f2be7b None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:01.048006 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5705c2-4802-449b-978f-75b436f2be7b None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:01.048307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5705c2-4802-449b-978f-75b436f2be7b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:01.049660 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5705c2-4802-449b-978f-75b436f2be7b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:01.059320 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 314/632] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:06:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:01.060495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae61c65-9bd7-4bc9-aba0-f04e5149595c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:01.060780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae61c65-9bd7-4bc9-aba0-f04e5149595c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:01.061055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae61c65-9bd7-4bc9-aba0-f04e5149595c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:01.065049 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f055b7-7537-45c5-821a-3bf774c34bad None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:01.078698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6ae61c65-9bd7-4bc9-aba0-f04e5149595c None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:01.079915 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6ae61c65-9bd7-4bc9-aba0-f04e5149595c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:01.098675 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916d1a61-8f0c-4afc-9bb0-69389c82fc64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:01.099355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916d1a61-8f0c-4afc-9bb0-69389c82fc64 None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:01.099631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916d1a61-8f0c-4afc-9bb0-69389c82fc64 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:01.099901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916d1a61-8f0c-4afc-9bb0-69389c82fc64 None admin] PATH_INFO: `/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/e0d7d8ed4ede4ddca192fce0a497689d/roles/71b1e7c4b5af44ddbb7a517760ad0119` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:01.114354 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916d1a61-8f0c-4afc-9bb0-69389c82fc64 None admin] RBAC: Authorizing `identity:create_grant(project_id=e5c1519fdae742f4a25eb7cd5476a8cb, user_id=e0d7d8ed4ede4ddca192fce0a497689d, role_id=71b1e7c4b5af44ddbb7a517760ad0119)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:01.115806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916d1a61-8f0c-4afc-9bb0-69389c82fc64 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:01.127583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:01.128511 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 319/633] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:06:00 2022] PUT /identity/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/e0d7d8ed4ede4ddca192fce0a497689d/roles/71b1e7c4b5af44ddbb7a517760ad0119 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:06:01.130031 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ac5d8c-6c38-4438-bdc8-3adcbe5a71ff None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:01.130314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ac5d8c-6c38-4438-bdc8-3adcbe5a71ff None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:01.130591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ac5d8c-6c38-4438-bdc8-3adcbe5a71ff None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:01.132507 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 320/634] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:01.133131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ac5d8c-6c38-4438-bdc8-3adcbe5a71ff None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:01.166884 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 315/635] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:06:01.166884 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae61c65-9bd7-4bc9-aba0-f04e5149595c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:01.273201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ac5d8c-6c38-4438-bdc8-3adcbe5a71ff None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:01.284642 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:01.328298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae61c65-9bd7-4bc9-aba0-f04e5149595c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:01.333594 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bb7c80-c0d9-4440-8bca-7ca584119a51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:01.334206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bb7c80-c0d9-4440-8bca-7ca584119a51 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:01.334478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bb7c80-c0d9-4440-8bca-7ca584119a51 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:01.334745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bb7c80-c0d9-4440-8bca-7ca584119a51 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:01.365916 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2868a83f-a3c6-41d0-93ab-9a468d7d0ac6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:01.367096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2868a83f-a3c6-41d0-93ab-9a468d7d0ac6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:01.367720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2868a83f-a3c6-41d0-93ab-9a468d7d0ac6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:01.368123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2868a83f-a3c6-41d0-93ab-9a468d7d0ac6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:01.380130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2868a83f-a3c6-41d0-93ab-9a468d7d0ac6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:01.381547 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2868a83f-a3c6-41d0-93ab-9a468d7d0ac6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:01.397342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-2868a83f-a3c6-41d0-93ab-9a468d7d0ac6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:01.397342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:01.397342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:01.397342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:01.397342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:01.397342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:01.397342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:01.397342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:01.397342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:01.397342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:01.397342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:01.397342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:01.397342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:01.397342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:01.397342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:01.397342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:01.398833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:01.398833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:01.398833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:01.398833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:01.398833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:01.398833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:01.398833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:01.398833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:01.398833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:01.398833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:01.398833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:01.398833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:01.398833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:01.398833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:01.398833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:01.398833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:01.398833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:01.400488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:01.400488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:01.400488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:01.400488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 316/636] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:01.400488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebb2d04-d2e3-4f1e-a4b9-0619d504f483 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:01.400790 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebb2d04-d2e3-4f1e-a4b9-0619d504f483 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:01.401018 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebb2d04-d2e3-4f1e-a4b9-0619d504f483 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:01.424121 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7ebb2d04-d2e3-4f1e-a4b9-0619d504f483 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:01.425320 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7ebb2d04-d2e3-4f1e-a4b9-0619d504f483 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:01.443145 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45bb7c80-c0d9-4440-8bca-7ca584119a51 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:01.445279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45bb7c80-c0d9-4440-8bca-7ca584119a51 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:01.452966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:01.498319 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 317/637] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:06:01.499854 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebb2d04-d2e3-4f1e-a4b9-0619d504f483 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:01.566141 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 321/638] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:06:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:06:01.567425 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bb7c80-c0d9-4440-8bca-7ca584119a51 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:01.579206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:01.620601 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebb2d04-d2e3-4f1e-a4b9-0619d504f483 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:01.664136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4c4bc3-1ffb-477c-b80f-7df37d5a121a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:01.665422 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4c4bc3-1ffb-477c-b80f-7df37d5a121a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:01.665912 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4c4bc3-1ffb-477c-b80f-7df37d5a121a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:01.666651 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4c4bc3-1ffb-477c-b80f-7df37d5a121a None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:01.668001 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4c4bc3-1ffb-477c-b80f-7df37d5a121a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:01.668761 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4c4bc3-1ffb-477c-b80f-7df37d5a121a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:01.671026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4c4bc3-1ffb-477c-b80f-7df37d5a121a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:01.690903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bb7c80-c0d9-4440-8bca-7ca584119a51 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:01.691456 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 318/639] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:01.692741 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4c4bc3-1ffb-477c-b80f-7df37d5a121a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:01.723230 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:01.738155 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989e0379-0dcf-415d-887f-cdb3248ed51b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:01.739636 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989e0379-0dcf-415d-887f-cdb3248ed51b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:01.740609 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989e0379-0dcf-415d-887f-cdb3248ed51b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:01.740884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989e0379-0dcf-415d-887f-cdb3248ed51b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:01.756407 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989e0379-0dcf-415d-887f-cdb3248ed51b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:01.761701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-989e0379-0dcf-415d-887f-cdb3248ed51b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:06:01.761701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:01.761701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:01.761701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:01.761701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:01.761701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:01.761701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:01.761701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:01.761701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:01.761701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:01.761701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:01.761701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:01.761701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:01.761701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:01.761701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:01.761701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:01.762775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:01.762775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:01.762775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:01.762775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:01.762775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:01.762775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:01.762775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:01.762775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:01.762775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:01.762775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:01.762775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:01.762775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:01.762775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:01.762775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:01.762775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:01.762775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:01.763633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:01.763633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:01.763633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:01.763633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:01.763633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:01.763633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:01.763633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:01.763633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:01.763633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:01.763633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:01.763633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:06:01.763633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:01.767497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 322/640] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:06:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:01.768318 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989e0379-0dcf-415d-887f-cdb3248ed51b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:01.809722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4c4bc3-1ffb-477c-b80f-7df37d5a121a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:01.840360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:01.848852 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60417328-fb01-4cae-ad01-4d887a601571 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:01.849486 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60417328-fb01-4cae-ad01-4d887a601571 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:01.850104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60417328-fb01-4cae-ad01-4d887a601571 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:01.850104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60417328-fb01-4cae-ad01-4d887a601571 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:01.862662 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60417328-fb01-4cae-ad01-4d887a601571 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:01.864622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60417328-fb01-4cae-ad01-4d887a601571 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:01.876744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989e0379-0dcf-415d-887f-cdb3248ed51b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:01.879606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-60417328-fb01-4cae-ad01-4d887a601571 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:01.879606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:01.879606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:01.879606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:01.879606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:01.879606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:01.879606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:01.879606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:01.879606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:01.879606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:01.879606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:01.879606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:01.879606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:01.879606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:01.879606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:01.879606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:01.880511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:01.880511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:01.880511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:01.880511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:01.880511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:01.880511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:01.880511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:01.880511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:01.880511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:01.880511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:01.880511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:01.880511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:01.880511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:01.880511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:01.880511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:01.880511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:01.880511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:01.881412 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:01.881412 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:01.881412 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:01.881412 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 319/641] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:01.881642 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60417328-fb01-4cae-ad01-4d887a601571 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:01.918433 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847fbb1d-d0b2-470f-9dd9-0191edcdab64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:01.919577 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847fbb1d-d0b2-470f-9dd9-0191edcdab64 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:01.919577 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847fbb1d-d0b2-470f-9dd9-0191edcdab64 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:01.919754 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847fbb1d-d0b2-470f-9dd9-0191edcdab64 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:01.926794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847fbb1d-d0b2-470f-9dd9-0191edcdab64 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:01.928274 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847fbb1d-d0b2-470f-9dd9-0191edcdab64 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:01.940105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-847fbb1d-d0b2-470f-9dd9-0191edcdab64 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 13 01:06:01.940105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:01.940105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:01.940105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:01.940105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:01.940105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:01.940105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:01.940105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:01.940105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:01.940105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:01.940105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:01.940105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:01.940105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:01.940105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:01.940105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:01.940105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:01.941276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:01.941276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:01.941276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:01.941276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:01.941276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:01.941276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:01.941276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:01.941276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:01.941276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:01.941276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:01.941276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:01.941276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:01.941276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:01.941276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:01.941276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:01.941276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:01.941276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:01.941276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:01.942229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:01.942229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:01.942229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:01.942229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:01.942229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:01.942229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:01.942229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:01.942229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:01.942229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 13 01:06:01.942229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:01.942229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 323/642] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:06:01 2022] GET /identity/v3/users/glance => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:01.942229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847fbb1d-d0b2-470f-9dd9-0191edcdab64 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:01.968612 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:01.992665 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60417328-fb01-4cae-ad01-4d887a601571 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:02.027901 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1ec581-9b91-4bfd-aefd-f2bcdfe3f260 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:02.028791 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1ec581-9b91-4bfd-aefd-f2bcdfe3f260 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:02.029085 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1ec581-9b91-4bfd-aefd-f2bcdfe3f260 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:02.029358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1ec581-9b91-4bfd-aefd-f2bcdfe3f260 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:02.041070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1ec581-9b91-4bfd-aefd-f2bcdfe3f260 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:02.046280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-bd1ec581-9b91-4bfd-aefd-f2bcdfe3f260 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:06:02.046280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:02.046280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:02.046280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:02.046280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:02.046280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:02.046280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:02.046280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:02.046280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:02.046280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:02.046280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:02.046280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:02.046280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:02.046280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:02.046280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:02.046280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:02.047298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:02.047298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:02.047298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:02.047298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:02.047298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:02.047298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:02.047298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:02.047298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:02.047298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:02.047298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:02.047298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:02.047298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:02.047298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:02.047298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:02.047298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:02.047298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:02.048186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:02.048186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:02.048186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:02.048186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:02.048186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:02.048186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:02.048186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:02.048186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:02.048186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:02.048186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:02.048186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:06:02.048186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:02.048186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 320/643] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:06:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:02.048186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1ec581-9b91-4bfd-aefd-f2bcdfe3f260 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:02.059121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847fbb1d-d0b2-470f-9dd9-0191edcdab64 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:02.089794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:02.098592 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fb9b8d-00dc-4f63-a560-87f283a6f6c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:02.099237 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fb9b8d-00dc-4f63-a560-87f283a6f6c4 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:02.099602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fb9b8d-00dc-4f63-a560-87f283a6f6c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:02.100076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fb9b8d-00dc-4f63-a560-87f283a6f6c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:02.100651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fb9b8d-00dc-4f63-a560-87f283a6f6c4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:02.100958 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fb9b8d-00dc-4f63-a560-87f283a6f6c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:02.102297 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fb9b8d-00dc-4f63-a560-87f283a6f6c4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:02.115730 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 324/644] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:06:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:02.115730 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fb9b8d-00dc-4f63-a560-87f283a6f6c4 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:02.153961 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1ec581-9b91-4bfd-aefd-f2bcdfe3f260 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:02.187944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecff61e5-c658-49d3-9ab2-4dc065e2a0be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:02.188892 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecff61e5-c658-49d3-9ab2-4dc065e2a0be None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:02.189167 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecff61e5-c658-49d3-9ab2-4dc065e2a0be None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:02.189436 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecff61e5-c658-49d3-9ab2-4dc065e2a0be None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:02.190060 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecff61e5-c658-49d3-9ab2-4dc065e2a0be None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:02.190356 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecff61e5-c658-49d3-9ab2-4dc065e2a0be None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:02.191724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecff61e5-c658-49d3-9ab2-4dc065e2a0be None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:02.193997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:02.208363 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 321/645] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:02.208363 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecff61e5-c658-49d3-9ab2-4dc065e2a0be None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:02.226952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fb9b8d-00dc-4f63-a560-87f283a6f6c4 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:02.267896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5ff459-2188-461c-b4ae-704d741c5a91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:02.268502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5ff459-2188-461c-b4ae-704d741c5a91 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:02.268778 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5ff459-2188-461c-b4ae-704d741c5a91 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:02.269047 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5ff459-2188-461c-b4ae-704d741c5a91 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:02.269772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5ff459-2188-461c-b4ae-704d741c5a91 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:02.270064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5ff459-2188-461c-b4ae-704d741c5a91 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:02.271582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5ff459-2188-461c-b4ae-704d741c5a91 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:02.289847 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 325/646] 10.222.0.15 () {56 vars in 1137 bytes} [Tue Sep 13 01:06:02 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:02.290644 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5ff459-2188-461c-b4ae-704d741c5a91 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:02.310420 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecff61e5-c658-49d3-9ab2-4dc065e2a0be None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:02.344399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:02.346056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e958427-83d7-486d-9ce0-7fa9a5d95fe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:02.346823 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e958427-83d7-486d-9ce0-7fa9a5d95fe3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:02.347029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e958427-83d7-486d-9ce0-7fa9a5d95fe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:02.347725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e958427-83d7-486d-9ce0-7fa9a5d95fe3 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:02.347994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e958427-83d7-486d-9ce0-7fa9a5d95fe3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:02.348353 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e958427-83d7-486d-9ce0-7fa9a5d95fe3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:02.349644 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e958427-83d7-486d-9ce0-7fa9a5d95fe3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:02.358243 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 322/647] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:06:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:02.359068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e958427-83d7-486d-9ce0-7fa9a5d95fe3 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:02.415613 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5ff459-2188-461c-b4ae-704d741c5a91 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:02.422562 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:02.460049 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387b869c-f562-46d7-8ef7-791efd3b634d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:02.460686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387b869c-f562-46d7-8ef7-791efd3b634d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:02.461021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387b869c-f562-46d7-8ef7-791efd3b634d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:02.461229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387b869c-f562-46d7-8ef7-791efd3b634d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:02.467966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e958427-83d7-486d-9ce0-7fa9a5d95fe3 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:02.471696 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387b869c-f562-46d7-8ef7-791efd3b634d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:02.473134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387b869c-f562-46d7-8ef7-791efd3b634d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:02.488503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-387b869c-f562-46d7-8ef7-791efd3b634d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:02.488503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:02.488503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:02.488503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:02.488503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:02.488503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:02.488503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:02.488503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:02.488503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:02.488503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:02.488503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:02.488503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:02.488503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:02.488503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:02.488503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:02.488503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:02.489645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:02.489645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:02.489645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:02.489645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:02.489645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:02.489645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:02.489645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:02.489645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:02.489645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:02.489645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:02.489645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:02.489645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:02.489645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:02.489645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:02.489645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:02.489645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:02.489645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:02.490530 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:02.490530 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:02.490530 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:02.490530 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 326/648] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:02.490753 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387b869c-f562-46d7-8ef7-791efd3b634d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:02.509016 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd5d479-a32f-450d-9a08-8a39ff040608 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:02.509651 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd5d479-a32f-450d-9a08-8a39ff040608 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:02.509928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd5d479-a32f-450d-9a08-8a39ff040608 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:02.510201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd5d479-a32f-450d-9a08-8a39ff040608 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:02.516842 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd5d479-a32f-450d-9a08-8a39ff040608 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:02.518361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd5d479-a32f-450d-9a08-8a39ff040608 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:02.529972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-7bd5d479-a32f-450d-9a08-8a39ff040608 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:02.529972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:02.529972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:02.529972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:02.529972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:02.529972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:02.529972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:02.529972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:02.529972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:02.529972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:02.529972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:02.529972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:02.529972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:02.529972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:02.529972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:02.530833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:02.530833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:02.530833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:02.530833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:02.530833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:02.530833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:02.530833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:02.530833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:02.530833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:02.530833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:02.530833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:02.530833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:02.530833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:02.530833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:02.530833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:02.530833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:02.530833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:02.532010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:02.532010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:02.532010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:02.532010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:02.532010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:02.532010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:02.532010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:02.532010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:02.532010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 323/649] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:02.532515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd5d479-a32f-450d-9a08-8a39ff040608 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:02.600086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:02.603797 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387b869c-f562-46d7-8ef7-791efd3b634d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:02.640883 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5474993c-b9b0-47e3-9c2d-544c0d1051ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:02.641494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5474993c-b9b0-47e3-9c2d-544c0d1051ce None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:02.641775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5474993c-b9b0-47e3-9c2d-544c0d1051ce None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:02.642003 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd5d479-a32f-450d-9a08-8a39ff040608 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:02.642109 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5474993c-b9b0-47e3-9c2d-544c0d1051ce None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:02.661509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:02.682356 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee939e4-f76a-4196-9545-056a44628db9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:02.683020 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee939e4-f76a-4196-9545-056a44628db9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:02.683302 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee939e4-f76a-4196-9545-056a44628db9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:02.683574 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee939e4-f76a-4196-9545-056a44628db9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:02.694280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee939e4-f76a-4196-9545-056a44628db9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:02.695775 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee939e4-f76a-4196-9545-056a44628db9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:02.710774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-5ee939e4-f76a-4196-9545-056a44628db9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:02.710774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:02.710774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:02.710774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:02.710774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:02.710774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:02.710774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:02.710774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:02.710774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:02.710774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:02.710774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:02.710774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:02.710774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:02.710774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:02.710774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:02.710774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:02.711738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:02.711738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:02.711738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:02.711738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:02.711738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:02.711738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:02.711738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:02.711738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:02.711738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:02.711738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:02.711738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:02.711738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:02.711738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:02.711738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:02.711738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:02.711738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:02.711738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:02.712604 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:02.712604 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:02.712604 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:02.712604 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 324/650] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:02.712834 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee939e4-f76a-4196-9545-056a44628db9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:02.723660 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5474993c-b9b0-47e3-9c2d-544c0d1051ce None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:02.725839 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5474993c-b9b0-47e3-9c2d-544c0d1051ce None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:02.822141 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee939e4-f76a-4196-9545-056a44628db9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:02.831444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:02.839729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 327/651] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:06:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:06:02.839729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5474993c-b9b0-47e3-9c2d-544c0d1051ce None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:02.863139 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c398460b-9ccd-4e01-b59d-8225600a915c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:02.863681 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398460b-9ccd-4e01-b59d-8225600a915c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:02.863956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398460b-9ccd-4e01-b59d-8225600a915c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:02.864231 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398460b-9ccd-4e01-b59d-8225600a915c None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:02.864868 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c398460b-9ccd-4e01-b59d-8225600a915c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:02.865173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c398460b-9ccd-4e01-b59d-8225600a915c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:02.867088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c398460b-9ccd-4e01-b59d-8225600a915c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:02.880001 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 325/652] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:02.880815 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c398460b-9ccd-4e01-b59d-8225600a915c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:02.911205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:02.949793 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5474993c-b9b0-47e3-9c2d-544c0d1051ce None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:02.986132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41ed4d0-ade7-4958-87c0-c771580dec87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:02.986614 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c398460b-9ccd-4e01-b59d-8225600a915c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:02.986836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41ed4d0-ade7-4958-87c0-c771580dec87 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:02.987033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41ed4d0-ade7-4958-87c0-c771580dec87 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:02.987310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41ed4d0-ade7-4958-87c0-c771580dec87 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:02.987996 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41ed4d0-ade7-4958-87c0-c771580dec87 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:02.988306 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41ed4d0-ade7-4958-87c0-c771580dec87 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:02.989719 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41ed4d0-ade7-4958-87c0-c771580dec87 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:03.002929 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 328/653] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:03.003603 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41ed4d0-ade7-4958-87c0-c771580dec87 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:03.027369 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7edd46-e66c-495a-ade7-b3711e96ceb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:03.027987 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7edd46-e66c-495a-ade7-b3711e96ceb1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:03.028364 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7edd46-e66c-495a-ade7-b3711e96ceb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:03.028564 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7edd46-e66c-495a-ade7-b3711e96ceb1 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:03.029295 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7edd46-e66c-495a-ade7-b3711e96ceb1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:03.029620 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7edd46-e66c-495a-ade7-b3711e96ceb1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:03.031029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7edd46-e66c-495a-ade7-b3711e96ceb1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:03.048123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 326/654] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:03.048123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7edd46-e66c-495a-ade7-b3711e96ceb1 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:03.068702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:03.115872 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41ed4d0-ade7-4958-87c0-c771580dec87 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:03.143420 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:03.153505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7edd46-e66c-495a-ade7-b3711e96ceb1 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:03.153676 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401880ef-ac53-4e0f-b1cb-4456129134f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:03.154236 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401880ef-ac53-4e0f-b1cb-4456129134f8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:03.154496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401880ef-ac53-4e0f-b1cb-4456129134f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:03.154765 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401880ef-ac53-4e0f-b1cb-4456129134f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:03.166054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401880ef-ac53-4e0f-b1cb-4456129134f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:03.167577 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401880ef-ac53-4e0f-b1cb-4456129134f8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:03.182135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-401880ef-ac53-4e0f-b1cb-4456129134f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:03.182135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:03.182135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:03.182135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:03.182135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:03.182135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:03.182135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:03.182135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:03.182135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:03.182135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:03.182135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:03.182135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:03.182135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:03.182135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:03.182135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:03.182135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:03.183149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:03.183149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:03.183149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:03.183149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:03.183149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:03.183149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:03.183149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:03.183149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:03.183149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:03.183149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:03.183149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:03.183149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:03.183149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:03.183149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:03.183149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:03.183149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:03.183149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:03.184055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:03.184055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:03.184055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:03.188355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 329/655] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:03.189124 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401880ef-ac53-4e0f-b1cb-4456129134f8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:03.195208 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a45a00a-4190-464a-875a-b9deef0dfdbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:03.195878 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a45a00a-4190-464a-875a-b9deef0dfdbe None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:03.196155 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a45a00a-4190-464a-875a-b9deef0dfdbe None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:03.196425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a45a00a-4190-464a-875a-b9deef0dfdbe None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:03.206084 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a45a00a-4190-464a-875a-b9deef0dfdbe None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:03.207616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a45a00a-4190-464a-875a-b9deef0dfdbe None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:03.220094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-0a45a00a-4190-464a-875a-b9deef0dfdbe None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 13 01:06:03.220094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:03.220094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:03.220094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:03.220094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:03.220094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:03.220094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:03.220094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:03.220094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:03.220094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:03.220094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:03.220094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:03.220094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:03.220094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:03.220094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:03.220094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:03.221131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:03.221131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:03.221131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:03.221131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:03.221131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:03.221131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:03.221131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:03.221131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:03.221131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:03.221131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:03.221131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:03.221131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:03.221131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:03.221131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:03.221131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:03.221131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:03.221131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:03.221131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:03.222072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:03.222072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:03.222072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:03.222072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:03.222072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:03.222072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:03.222072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:03.222072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:03.222072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 13 01:06:03.222072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:03.222072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 327/656] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:06:03 2022] GET /identity/v3/users/neutron => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:03.222072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a45a00a-4190-464a-875a-b9deef0dfdbe None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:03.304281 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401880ef-ac53-4e0f-b1cb-4456129134f8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:03.317685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:03.335144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a45a00a-4190-464a-875a-b9deef0dfdbe None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:03.339173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8028c9bb-f114-4be7-bc70-99d826748539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:03.339770 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8028c9bb-f114-4be7-bc70-99d826748539 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:03.340043 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8028c9bb-f114-4be7-bc70-99d826748539 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:03.340308 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8028c9bb-f114-4be7-bc70-99d826748539 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:03.341011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8028c9bb-f114-4be7-bc70-99d826748539 None admin] RBAC: Adding query filter params (role.id=71b1e7c4b5af44ddbb7a517760ad0119, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=2f2da4d2d3604d16b58dc25b8c17c49a) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:03.341339 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8028c9bb-f114-4be7-bc70-99d826748539 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:03.342739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8028c9bb-f114-4be7-bc70-99d826748539 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:03.348296 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 330/657] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:06:03 2022] GET /identity/v3/role_assignments?user.id=2f2da4d2d3604d16b58dc25b8c17c49a&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=71b1e7c4b5af44ddbb7a517760ad0119 => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:03.349051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8028c9bb-f114-4be7-bc70-99d826748539 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:03.381939 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a73e72-7b23-48b1-a5b2-6a52bef7c060 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:03.382502 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a73e72-7b23-48b1-a5b2-6a52bef7c060 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:03.382784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a73e72-7b23-48b1-a5b2-6a52bef7c060 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:03.383088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a73e72-7b23-48b1-a5b2-6a52bef7c060 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:03.391862 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a73e72-7b23-48b1-a5b2-6a52bef7c060 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:03.393331 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a73e72-7b23-48b1-a5b2-6a52bef7c060 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:03.406308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-14a73e72-7b23-48b1-a5b2-6a52bef7c060 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 01:06:03.406308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:03.406308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:03.406308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:03.406308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:03.406308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:03.406308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:03.406308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:03.406308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:03.406308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:03.406308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:03.406308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:03.406308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:03.406308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:03.406308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:03.406308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:03.407354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:03.407354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:03.407354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:03.407354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:03.407354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:03.407354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:03.407354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:03.407354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:03.407354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:03.407354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:03.407354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:03.407354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:03.407354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:03.407354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:03.407354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:03.407354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:03.407354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:03.407354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:03.408882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:03.408882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:03.408882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:03.408882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:03.408882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:03.408882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:03.408882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:03.408882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:03.408882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 01:06:03.408882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:03.408882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 328/658] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:06:03 2022] GET /identity/v3/users/cinder => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:03.408882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a73e72-7b23-48b1-a5b2-6a52bef7c060 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:03.425916 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:03.472120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8028c9bb-f114-4be7-bc70-99d826748539 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:03.510138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6dbdc0-f6a3-4cac-a51d-a777064e34c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:03.510753 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6dbdc0-f6a3-4cac-a51d-a777064e34c0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:03.511056 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6dbdc0-f6a3-4cac-a51d-a777064e34c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:03.511328 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6dbdc0-f6a3-4cac-a51d-a777064e34c0 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:03.511968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6dbdc0-f6a3-4cac-a51d-a777064e34c0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:03.512266 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6dbdc0-f6a3-4cac-a51d-a777064e34c0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:03.513627 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6dbdc0-f6a3-4cac-a51d-a777064e34c0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:03.527970 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a73e72-7b23-48b1-a5b2-6a52bef7c060 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:03.528756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 331/659] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:03.529962 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68cfae3-fb7f-4f2b-8114-5e3d0668acc0 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:03.530245 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68cfae3-fb7f-4f2b-8114-5e3d0668acc0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:03.530511 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68cfae3-fb7f-4f2b-8114-5e3d0668acc0 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:03.535936 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 332/660] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:03.535936 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68cfae3-fb7f-4f2b-8114-5e3d0668acc0 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:03.573496 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27a5b52-4e2b-4bba-87ec-4460b5b6eaf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:03.574201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27a5b52-4e2b-4bba-87ec-4460b5b6eaf0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:03.574376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27a5b52-4e2b-4bba-87ec-4460b5b6eaf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:03.574642 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27a5b52-4e2b-4bba-87ec-4460b5b6eaf0 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:03.576810 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27a5b52-4e2b-4bba-87ec-4460b5b6eaf0 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:03.576810 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27a5b52-4e2b-4bba-87ec-4460b5b6eaf0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:03.577297 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:03.577381 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27a5b52-4e2b-4bba-87ec-4460b5b6eaf0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:03.599603 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 329/661] 10.222.0.15 () {56 vars in 1139 bytes} [Tue Sep 13 01:06:03 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:03.600409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27a5b52-4e2b-4bba-87ec-4460b5b6eaf0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:03.652110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68cfae3-fb7f-4f2b-8114-5e3d0668acc0 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:03.669799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:03.689519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01776c59-6915-4468-8435-0a0d7df31db4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:03.690214 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01776c59-6915-4468-8435-0a0d7df31db4 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:03.690494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01776c59-6915-4468-8435-0a0d7df31db4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:03.690763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01776c59-6915-4468-8435-0a0d7df31db4 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:03.691541 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01776c59-6915-4468-8435-0a0d7df31db4 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:03.691826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01776c59-6915-4468-8435-0a0d7df31db4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:03.693237 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01776c59-6915-4468-8435-0a0d7df31db4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:03.704855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27a5b52-4e2b-4bba-87ec-4460b5b6eaf0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:03.715750 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 333/662] 10.222.0.15 () {56 vars in 1137 bytes} [Tue Sep 13 01:06:03 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:03.715750 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fe9771-71f9-437a-8880-8d03cb57eeea None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:03.715750 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fe9771-71f9-437a-8880-8d03cb57eeea None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:03.715750 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fe9771-71f9-437a-8880-8d03cb57eeea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:03.732528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-20fe9771-71f9-437a-8880-8d03cb57eeea None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:03.733711 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-20fe9771-71f9-437a-8880-8d03cb57eeea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:03.739661 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1c5006-5ab6-4ce6-81f3-1d5ce0034903 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:03.740289 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1c5006-5ab6-4ce6-81f3-1d5ce0034903 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:03.740565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1c5006-5ab6-4ce6-81f3-1d5ce0034903 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:03.740833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1c5006-5ab6-4ce6-81f3-1d5ce0034903 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:03.748192 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1c5006-5ab6-4ce6-81f3-1d5ce0034903 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:03.749609 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1c5006-5ab6-4ce6-81f3-1d5ce0034903 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:03.760510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-7e1c5006-5ab6-4ce6-81f3-1d5ce0034903 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:03.760510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:03.760510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:03.760510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:03.760510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:03.760510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:03.760510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:03.760510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:03.760510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:03.760510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:03.760510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:03.760510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:03.760510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:03.760510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:03.760510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:03.761416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:03.761416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:03.761416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:03.761416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:03.761416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:03.761416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:03.761416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:03.761416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:03.761416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:03.761416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:03.761416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:03.761416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:03.761416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:03.761416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:03.761416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:03.761416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:03.761416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:03.762290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:03.762290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:03.762290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:03.762290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:03.762290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:03.762290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:03.762290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:03.762290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:03.762290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 330/663] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:03.762778 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1c5006-5ab6-4ce6-81f3-1d5ce0034903 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:03.814736 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 334/664] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:06:03.815712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fe9771-71f9-437a-8880-8d03cb57eeea None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:03.841282 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:03.896998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1c5006-5ab6-4ce6-81f3-1d5ce0034903 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:03.919489 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fe9771-71f9-437a-8880-8d03cb57eeea None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:03.927373 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:03.936629 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a14b5f6-4cab-407d-85f0-a210254dff9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:03.937605 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a14b5f6-4cab-407d-85f0-a210254dff9a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:03.938413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a14b5f6-4cab-407d-85f0-a210254dff9a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:03.938912 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a14b5f6-4cab-407d-85f0-a210254dff9a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:03.955471 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5b9cdb-dcd6-492b-b5ea-05566fc80331 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:03.956085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5b9cdb-dcd6-492b-b5ea-05566fc80331 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:03.956365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5b9cdb-dcd6-492b-b5ea-05566fc80331 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:03.956641 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5b9cdb-dcd6-492b-b5ea-05566fc80331 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:04.012821 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a14b5f6-4cab-407d-85f0-a210254dff9a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:04.014616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a14b5f6-4cab-407d-85f0-a210254dff9a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:04.027230 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5b9cdb-dcd6-492b-b5ea-05566fc80331 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:04.028918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5b9cdb-dcd6-492b-b5ea-05566fc80331 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:04.059465 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:04.122148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 331/665] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:06:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:06:04.123366 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a14b5f6-4cab-407d-85f0-a210254dff9a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:04.135436 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 335/666] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:06:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:06:04.140618 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5b9cdb-dcd6-492b-b5ea-05566fc80331 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:04.154112 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:04.229692 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a14b5f6-4cab-407d-85f0-a210254dff9a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:04.246251 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5b9cdb-dcd6-492b-b5ea-05566fc80331 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:04.263449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ecce26-ee76-4f17-91cd-4d21c630049a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:04.264182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ecce26-ee76-4f17-91cd-4d21c630049a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:04.264575 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ecce26-ee76-4f17-91cd-4d21c630049a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:04.264943 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ecce26-ee76-4f17-91cd-4d21c630049a None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:04.265722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ecce26-ee76-4f17-91cd-4d21c630049a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:04.266190 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ecce26-ee76-4f17-91cd-4d21c630049a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:04.267726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ecce26-ee76-4f17-91cd-4d21c630049a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:04.270301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:04.281053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 332/667] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:04.282203 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ecce26-ee76-4f17-91cd-4d21c630049a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:04.291999 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fd16e2-b791-4fbb-8175-7c183a804a6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:04.293234 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fd16e2-b791-4fbb-8175-7c183a804a6c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:04.293511 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fd16e2-b791-4fbb-8175-7c183a804a6c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:04.293781 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fd16e2-b791-4fbb-8175-7c183a804a6c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:04.306759 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fd16e2-b791-4fbb-8175-7c183a804a6c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:04.311523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-56fd16e2-b791-4fbb-8175-7c183a804a6c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:06:04.311523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:04.311523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:04.311523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:04.311523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:04.311523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:04.311523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:04.311523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:04.311523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:04.311523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:04.311523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:04.311523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:04.311523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:04.311523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:04.311523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:04.311523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:04.312658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:04.312658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:04.312658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:04.312658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:04.312658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:04.312658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:04.312658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:04.312658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:04.312658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:04.312658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:04.312658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:04.312658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:04.312658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:04.312658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:04.312658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:04.312658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:04.313980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:04.313980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:04.313980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:04.313980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:04.313980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:04.313980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:04.313980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:04.313980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:04.313980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:04.313980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:04.313980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:06:04.313980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:04.313980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 336/668] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:06:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:04.313980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fd16e2-b791-4fbb-8175-7c183a804a6c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:04.380278 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:04.394292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ecce26-ee76-4f17-91cd-4d21c630049a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:04.431104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fd16e2-b791-4fbb-8175-7c183a804a6c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:04.436977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650f56e0-1a7a-42b1-a639-fa8ddb898c48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:04.437905 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650f56e0-1a7a-42b1-a639-fa8ddb898c48 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:04.438439 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650f56e0-1a7a-42b1-a639-fa8ddb898c48 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:04.438713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650f56e0-1a7a-42b1-a639-fa8ddb898c48 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:04.451507 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650f56e0-1a7a-42b1-a639-fa8ddb898c48 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:04.453700 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650f56e0-1a7a-42b1-a639-fa8ddb898c48 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:04.469295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2e2313-3db5-4c34-90a5-a67c79461631 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:04.470029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2e2313-3db5-4c34-90a5-a67c79461631 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:04.470429 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2e2313-3db5-4c34-90a5-a67c79461631 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:04.470795 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2e2313-3db5-4c34-90a5-a67c79461631 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:04.472802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-650f56e0-1a7a-42b1-a639-fa8ddb898c48 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:04.472802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:04.472802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:04.472802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:04.472802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:04.472802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:04.472802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:04.472802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:04.472802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:04.472802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:04.472802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:04.472802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:04.472802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:04.472802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:04.472802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:04.472802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:04.473877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:04.473877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:04.473877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:04.473877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:04.473877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:04.473877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:04.473877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:04.473877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:04.473877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:04.473877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:04.473877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:04.473877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:04.473877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:04.473877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:04.473877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:04.473877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:04.473877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:04.479997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:04.479997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:04.479997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:04.479997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 333/669] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:04.479997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650f56e0-1a7a-42b1-a639-fa8ddb898c48 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:04.488086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2e2313-3db5-4c34-90a5-a67c79461631 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:04.489626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2e2313-3db5-4c34-90a5-a67c79461631 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:04.505179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-7d2e2313-3db5-4c34-90a5-a67c79461631 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:04.505179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:04.505179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:04.505179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:04.505179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:04.505179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:04.505179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:04.505179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:04.505179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:04.505179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:04.505179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:04.505179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:04.505179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:04.505179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:04.505179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:04.505179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:04.506326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:04.506326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:04.506326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:04.506326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:04.506326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:04.506326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:04.506326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:04.506326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:04.506326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:04.506326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:04.506326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:04.506326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:04.506326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:04.506326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:04.506326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:04.506326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:04.506326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:04.508055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:04.508055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:04.508055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:04.509257 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 337/670] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:04.509964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2e2313-3db5-4c34-90a5-a67c79461631 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:04.548090 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:04.593755 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650f56e0-1a7a-42b1-a639-fa8ddb898c48 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:04.617955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2e2313-3db5-4c34-90a5-a67c79461631 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:04.636821 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd38b5ab-5388-46d2-a17b-1c09b048554e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:04.637575 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd38b5ab-5388-46d2-a17b-1c09b048554e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:04.638465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd38b5ab-5388-46d2-a17b-1c09b048554e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:04.638846 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd38b5ab-5388-46d2-a17b-1c09b048554e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:04.639993 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd38b5ab-5388-46d2-a17b-1c09b048554e None admin] RBAC: Adding query filter params (role.id=71b1e7c4b5af44ddbb7a517760ad0119, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=eef98bcf2699461d9a7f3c3bffe82f34) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:04.640447 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd38b5ab-5388-46d2-a17b-1c09b048554e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:04.642826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd38b5ab-5388-46d2-a17b-1c09b048554e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:04.649874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 334/671] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:06:04 2022] GET /identity/v3/role_assignments?user.id=eef98bcf2699461d9a7f3c3bffe82f34&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=71b1e7c4b5af44ddbb7a517760ad0119 => generated 608 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:04.650772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd38b5ab-5388-46d2-a17b-1c09b048554e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:04.659940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:04.661374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e2ded0-df72-425b-bc73-49f3a722228e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:04.662389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e2ded0-df72-425b-bc73-49f3a722228e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:04.662683 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e2ded0-df72-425b-bc73-49f3a722228e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:04.663509 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e2ded0-df72-425b-bc73-49f3a722228e None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:04.664662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e2ded0-df72-425b-bc73-49f3a722228e None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:04.664987 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e2ded0-df72-425b-bc73-49f3a722228e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:04.666788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e2ded0-df72-425b-bc73-49f3a722228e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:04.682481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 338/672] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:06:04 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:04.683947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e2ded0-df72-425b-bc73-49f3a722228e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:04.765252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd38b5ab-5388-46d2-a17b-1c09b048554e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:04.804381 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35cb104-0c68-4b8b-8c30-4f1ea0ef3964 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:04.805103 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35cb104-0c68-4b8b-8c30-4f1ea0ef3964 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:04.805468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35cb104-0c68-4b8b-8c30-4f1ea0ef3964 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:04.805805 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35cb104-0c68-4b8b-8c30-4f1ea0ef3964 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:04.806613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35cb104-0c68-4b8b-8c30-4f1ea0ef3964 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:04.807024 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35cb104-0c68-4b8b-8c30-4f1ea0ef3964 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:04.808495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35cb104-0c68-4b8b-8c30-4f1ea0ef3964 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:04.822284 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:04.827819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 335/673] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:04.828734 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35cb104-0c68-4b8b-8c30-4f1ea0ef3964 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:04.829692 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e2ded0-df72-425b-bc73-49f3a722228e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:04.871183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72eb65f8-88f6-47c1-a5a6-4fcbfa26da4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:04.871816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72eb65f8-88f6-47c1-a5a6-4fcbfa26da4a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:04.872062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72eb65f8-88f6-47c1-a5a6-4fcbfa26da4a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:04.872332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72eb65f8-88f6-47c1-a5a6-4fcbfa26da4a None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:04.872973 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72eb65f8-88f6-47c1-a5a6-4fcbfa26da4a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:04.873276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72eb65f8-88f6-47c1-a5a6-4fcbfa26da4a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:04.874640 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72eb65f8-88f6-47c1-a5a6-4fcbfa26da4a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:04.890356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 339/674] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:04.893164 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72eb65f8-88f6-47c1-a5a6-4fcbfa26da4a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:04.921694 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:04.967645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35cb104-0c68-4b8b-8c30-4f1ea0ef3964 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:05.010693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f2a364-b664-4f8c-82f9-d02b38929993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:05.011352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f2a364-b664-4f8c-82f9-d02b38929993 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:05.011631 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f2a364-b664-4f8c-82f9-d02b38929993 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:05.011893 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f2a364-b664-4f8c-82f9-d02b38929993 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:05.016324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72eb65f8-88f6-47c1-a5a6-4fcbfa26da4a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:05.023395 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f2a364-b664-4f8c-82f9-d02b38929993 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:05.025633 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f2a364-b664-4f8c-82f9-d02b38929993 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:05.052520 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f912dd18-2a59-472e-995c-9f6b903c2b6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:05.053154 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f912dd18-2a59-472e-995c-9f6b903c2b6c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:05.053429 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f912dd18-2a59-472e-995c-9f6b903c2b6c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:05.054230 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f912dd18-2a59-472e-995c-9f6b903c2b6c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:05.055215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-05f2a364-b664-4f8c-82f9-d02b38929993 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:05.055215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:05.055215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:05.055215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:05.055215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:05.055215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:05.055215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:05.055215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:05.055215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:05.055215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:05.055215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:05.055215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:05.055215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:05.055215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:05.055215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:05.055215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:05.056142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:05.056142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:05.056142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:05.056142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:05.056142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:05.056142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:05.056142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:05.056142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:05.056142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:05.056142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:05.056142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:05.056142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:05.056142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:05.056142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:05.056142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:05.056142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:05.056142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:05.057046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:05.057046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:05.057046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:05.057597 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 336/675] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:05.062884 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f2a364-b664-4f8c-82f9-d02b38929993 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:05.066162 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f912dd18-2a59-472e-995c-9f6b903c2b6c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:05.067637 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f912dd18-2a59-472e-995c-9f6b903c2b6c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:05.080094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-f912dd18-2a59-472e-995c-9f6b903c2b6c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:05.080094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:05.080094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:05.080094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:05.080094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:05.080094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:05.080094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:05.080094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:05.080094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:05.080094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:05.080094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:05.080094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:05.080094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:05.080094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:05.080094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:05.081037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:05.081037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:05.081037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:05.081037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:05.081037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:05.081037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:05.081037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:05.081037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:05.081037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:05.081037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:05.081037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:05.081037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:05.081037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:05.081037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:05.081037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:05.081037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:05.081037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:05.081942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:05.081942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:05.081942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:05.081942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:05.081942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:05.081942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:05.081942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:05.081942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:05.081942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 340/676] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:05.082428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f912dd18-2a59-472e-995c-9f6b903c2b6c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:05.108439 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:05.184237 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f2a364-b664-4f8c-82f9-d02b38929993 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:05.201743 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:05.221525 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041620ba-2a50-415e-b494-2312e55ef91a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:05.222549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f912dd18-2a59-472e-995c-9f6b903c2b6c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:05.222666 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041620ba-2a50-415e-b494-2312e55ef91a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:05.223150 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041620ba-2a50-415e-b494-2312e55ef91a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:05.223618 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041620ba-2a50-415e-b494-2312e55ef91a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:05.231442 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-041620ba-2a50-415e-b494-2312e55ef91a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:05.233237 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-041620ba-2a50-415e-b494-2312e55ef91a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:05.245249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-041620ba-2a50-415e-b494-2312e55ef91a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:05.245249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:05.245249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:05.245249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:05.245249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:05.245249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:05.245249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:05.245249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:05.245249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:05.245249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:05.245249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:05.245249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:05.245249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:05.245249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:05.245249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:05.246148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:05.246148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:05.246148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:05.246148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:05.246148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:05.246148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:05.246148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:05.246148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:05.246148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:05.246148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:05.246148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:05.246148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:05.246148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:05.246148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:05.246148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:05.246148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:05.246148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:05.247035 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:05.247035 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:05.247035 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:05.247035 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:05.247035 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:05.247035 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:05.247035 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:05.247035 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:05.249411 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 337/677] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:05.250405 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041620ba-2a50-415e-b494-2312e55ef91a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:05.266284 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded11d49-e9f5-40b2-9e48-57e4bcbe350c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:05.267031 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded11d49-e9f5-40b2-9e48-57e4bcbe350c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:05.267248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded11d49-e9f5-40b2-9e48-57e4bcbe350c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:05.267524 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded11d49-e9f5-40b2-9e48-57e4bcbe350c None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:05.268580 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded11d49-e9f5-40b2-9e48-57e4bcbe350c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:05.268580 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded11d49-e9f5-40b2-9e48-57e4bcbe350c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:05.270012 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded11d49-e9f5-40b2-9e48-57e4bcbe350c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:05.286774 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 341/678] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:05.287630 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded11d49-e9f5-40b2-9e48-57e4bcbe350c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:05.370971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041620ba-2a50-415e-b494-2312e55ef91a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:05.390325 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:05.408111 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a85055-f7e7-4d4e-92c9-efcdea7d5841 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:05.408837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a85055-f7e7-4d4e-92c9-efcdea7d5841 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:05.409125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a85055-f7e7-4d4e-92c9-efcdea7d5841 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:05.409402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a85055-f7e7-4d4e-92c9-efcdea7d5841 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:05.410102 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a85055-f7e7-4d4e-92c9-efcdea7d5841 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:05.410409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a85055-f7e7-4d4e-92c9-efcdea7d5841 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:05.412096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a85055-f7e7-4d4e-92c9-efcdea7d5841 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:05.418737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded11d49-e9f5-40b2-9e48-57e4bcbe350c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:05.429227 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 338/679] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:05.430116 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a85055-f7e7-4d4e-92c9-efcdea7d5841 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:05.458413 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efbedde-e5d6-4ca1-bc92-61dcbbe64beb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:05.459364 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efbedde-e5d6-4ca1-bc92-61dcbbe64beb None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:05.459972 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efbedde-e5d6-4ca1-bc92-61dcbbe64beb None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:05.460251 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efbedde-e5d6-4ca1-bc92-61dcbbe64beb None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:05.460933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efbedde-e5d6-4ca1-bc92-61dcbbe64beb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:05.461238 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efbedde-e5d6-4ca1-bc92-61dcbbe64beb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:05.462646 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efbedde-e5d6-4ca1-bc92-61dcbbe64beb None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:05.468978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:05.475069 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 342/680] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:05.476555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efbedde-e5d6-4ca1-bc92-61dcbbe64beb None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:05.547359 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a85055-f7e7-4d4e-92c9-efcdea7d5841 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:05.585778 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fbccea-db00-46b0-959b-6c04a8c80605 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:05.586756 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fbccea-db00-46b0-959b-6c04a8c80605 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:05.587468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fbccea-db00-46b0-959b-6c04a8c80605 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:05.587909 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fbccea-db00-46b0-959b-6c04a8c80605 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:05.596398 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fbccea-db00-46b0-959b-6c04a8c80605 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:05.598315 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fbccea-db00-46b0-959b-6c04a8c80605 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:05.601020 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efbedde-e5d6-4ca1-bc92-61dcbbe64beb None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:05.612123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-58fbccea-db00-46b0-959b-6c04a8c80605 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 13 01:06:05.612123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:05.612123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:05.612123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:05.612123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:05.612123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:05.612123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:05.612123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:05.612123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:05.612123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:05.612123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:05.612123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:05.612123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:05.612123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:05.612123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:05.612123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:05.613283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:05.613283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:05.613283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:05.613283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:05.613283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:05.613283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:05.613283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:05.613283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:05.613283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:05.613283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:05.613283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:05.613283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:05.613283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:05.613283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:05.613283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:05.613283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:05.613283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:05.613283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:05.614409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:05.614409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:05.614409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:05.614409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:05.614409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:05.614409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:05.614409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:05.614409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:05.614409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 13 01:06:05.614409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:05.616943 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 339/681] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:06:05 2022] GET /identity/v3/users/swift => generated 83 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:05.617996 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fbccea-db00-46b0-959b-6c04a8c80605 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:05.643265 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67536d1-6d4e-4083-ba9e-31efc0b610e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:05.643874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67536d1-6d4e-4083-ba9e-31efc0b610e4 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:05.644155 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67536d1-6d4e-4083-ba9e-31efc0b610e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:05.644427 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67536d1-6d4e-4083-ba9e-31efc0b610e4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:05.645139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67536d1-6d4e-4083-ba9e-31efc0b610e4 None admin] RBAC: Adding query filter params (role.id=71b1e7c4b5af44ddbb7a517760ad0119, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=09cd263f72484733adaaa0850ebfa609) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:05.645451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67536d1-6d4e-4083-ba9e-31efc0b610e4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:05.646881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67536d1-6d4e-4083-ba9e-31efc0b610e4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:05.653336 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 343/682] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:06:05 2022] GET /identity/v3/role_assignments?user.id=09cd263f72484733adaaa0850ebfa609&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=71b1e7c4b5af44ddbb7a517760ad0119 => generated 608 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:05.654544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f44286-7c6e-467c-bf11-22a9f7c62c7b None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:05.654829 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f44286-7c6e-467c-bf11-22a9f7c62c7b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:05.655131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f44286-7c6e-467c-bf11-22a9f7c62c7b None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:05.659010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 344/683] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:05 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:05.660054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f44286-7c6e-467c-bf11-22a9f7c62c7b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:05.663776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:05.752369 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fbccea-db00-46b0-959b-6c04a8c80605 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:05.779246 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:05.791206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f44286-7c6e-467c-bf11-22a9f7c62c7b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:05.791444 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67427ebc-f5ad-4335-bd0e-5f9eb7ac4523 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:05.791931 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67427ebc-f5ad-4335-bd0e-5f9eb7ac4523 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:05.792215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67427ebc-f5ad-4335-bd0e-5f9eb7ac4523 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:05.792493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67427ebc-f5ad-4335-bd0e-5f9eb7ac4523 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:05.793228 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67427ebc-f5ad-4335-bd0e-5f9eb7ac4523 None admin] RBAC: Adding query filter params (role.id=71b1e7c4b5af44ddbb7a517760ad0119, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=f90a5e57daa643eab94c01e21fcbc4c4) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:05.793539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67427ebc-f5ad-4335-bd0e-5f9eb7ac4523 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:05.795131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67427ebc-f5ad-4335-bd0e-5f9eb7ac4523 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:05.802450 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 340/684] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:06:05 2022] GET /identity/v3/role_assignments?user.id=f90a5e57daa643eab94c01e21fcbc4c4&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=71b1e7c4b5af44ddbb7a517760ad0119 => generated 608 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:05.804323 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432d0cb9-3903-4de0-9791-555155d4ded0 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:05.804616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432d0cb9-3903-4de0-9791-555155d4ded0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:05.804892 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432d0cb9-3903-4de0-9791-555155d4ded0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:05.824487 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-432d0cb9-3903-4de0-9791-555155d4ded0 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:05.825728 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-432d0cb9-3903-4de0-9791-555155d4ded0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:05.833190 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819e938a-f7af-47df-8a30-d68be22535b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:05.833815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819e938a-f7af-47df-8a30-d68be22535b1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:05.834145 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819e938a-f7af-47df-8a30-d68be22535b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:05.834420 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819e938a-f7af-47df-8a30-d68be22535b1 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:05.835835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819e938a-f7af-47df-8a30-d68be22535b1 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:05.836144 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819e938a-f7af-47df-8a30-d68be22535b1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:05.837614 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819e938a-f7af-47df-8a30-d68be22535b1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:05.860191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 345/685] 10.222.0.15 () {56 vars in 1135 bytes} [Tue Sep 13 01:06:05 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:05.865427 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819e938a-f7af-47df-8a30-d68be22535b1 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:05.917401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 341/686] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:06:05.956031 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:06.004086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819e938a-f7af-47df-8a30-d68be22535b1 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:06.042174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001c7970-e31b-41df-950c-2854a754562a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:06.042789 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001c7970-e31b-41df-950c-2854a754562a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:06.043093 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001c7970-e31b-41df-950c-2854a754562a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:06.043360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001c7970-e31b-41df-950c-2854a754562a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:06.120603 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001c7970-e31b-41df-950c-2854a754562a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:06.122037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001c7970-e31b-41df-950c-2854a754562a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:06.181782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:06.206158 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432d0cb9-3903-4de0-9791-555155d4ded0 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:06.246817 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 346/687] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:06:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 13 01:06:06.252968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001c7970-e31b-41df-950c-2854a754562a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:06.396224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:06.410306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432d0cb9-3903-4de0-9791-555155d4ded0 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:06.426888 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001c7970-e31b-41df-950c-2854a754562a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:06.479799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9184909a-aba7-4e7f-b8d4-3ab51867ddbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:06.480773 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9184909a-aba7-4e7f-b8d4-3ab51867ddbf None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:06.481100 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9184909a-aba7-4e7f-b8d4-3ab51867ddbf None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:06.481371 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9184909a-aba7-4e7f-b8d4-3ab51867ddbf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:06.493439 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9184909a-aba7-4e7f-b8d4-3ab51867ddbf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:06.498505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-9184909a-aba7-4e7f-b8d4-3ab51867ddbf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:06:06.498505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:06.498505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:06.498505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:06.498505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:06.498505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:06.498505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:06.498505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:06.498505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:06.498505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:06.498505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:06.498505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:06.498505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:06.498505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:06.498505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:06.498505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:06.499524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:06.499524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:06.499524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:06.499524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:06.499524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:06.499524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:06.499524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:06.499524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:06.499524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:06.499524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:06.499524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:06.499524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:06.499524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:06.499524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:06.499524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:06.499524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:06.500360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:06.500360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:06.500360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:06.500360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:06.500360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:06.500360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:06.500360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:06.500360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:06.500360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:06.500360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:06.500360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:06:06.500360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:06.501013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c044931b-45cf-4903-a791-87b8133d79f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:06.501013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c044931b-45cf-4903-a791-87b8133d79f6 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:06.501013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c044931b-45cf-4903-a791-87b8133d79f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:06.501363 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c044931b-45cf-4903-a791-87b8133d79f6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:06.505990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 342/688] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:06:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:06.510571 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9184909a-aba7-4e7f-b8d4-3ab51867ddbf None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:06.518749 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c044931b-45cf-4903-a791-87b8133d79f6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:06.520347 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c044931b-45cf-4903-a791-87b8133d79f6 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:06.550150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-c044931b-45cf-4903-a791-87b8133d79f6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:06.550150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:06.550150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:06.550150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:06.550150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:06.550150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:06.550150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:06.550150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:06.550150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:06.550150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:06.550150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:06.550150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:06.550150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:06.550150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:06.550150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:06.550150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:06.551198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:06.551198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:06.551198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:06.551198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:06.551198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:06.551198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:06.551198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:06.551198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:06.551198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:06.551198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:06.551198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:06.551198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:06.551198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:06.551198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:06.551198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:06.551198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:06.551198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:06.552083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:06.552083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:06.552083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:06.552083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 347/689] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:06.563337 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c044931b-45cf-4903-a791-87b8133d79f6 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:06.590283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:06.669327 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9184909a-aba7-4e7f-b8d4-3ab51867ddbf None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:06.733538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7088f508-a048-46b2-99ac-fbd8ce26c064 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:06.734152 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7088f508-a048-46b2-99ac-fbd8ce26c064 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:06.734428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7088f508-a048-46b2-99ac-fbd8ce26c064 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:06.734697 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7088f508-a048-46b2-99ac-fbd8ce26c064 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:06.735599 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7088f508-a048-46b2-99ac-fbd8ce26c064 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:06.735735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7088f508-a048-46b2-99ac-fbd8ce26c064 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:06.737041 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7088f508-a048-46b2-99ac-fbd8ce26c064 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:06.744950 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c044931b-45cf-4903-a791-87b8133d79f6 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:06.755981 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 343/690] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:06:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:06.763083 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7088f508-a048-46b2-99ac-fbd8ce26c064 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:06.789296 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:06.821940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64406ca0-6d77-40e2-a4e8-8f429e0ea520 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:06.822558 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64406ca0-6d77-40e2-a4e8-8f429e0ea520 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:06.822904 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64406ca0-6d77-40e2-a4e8-8f429e0ea520 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:06.823185 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64406ca0-6d77-40e2-a4e8-8f429e0ea520 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:06.824558 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64406ca0-6d77-40e2-a4e8-8f429e0ea520 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:06.824872 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64406ca0-6d77-40e2-a4e8-8f429e0ea520 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:06.826241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64406ca0-6d77-40e2-a4e8-8f429e0ea520 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:06.852041 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 348/691] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:06.860061 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64406ca0-6d77-40e2-a4e8-8f429e0ea520 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:06.947902 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7088f508-a048-46b2-99ac-fbd8ce26c064 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:06.993270 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:07.027333 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a639e2cf-4e77-4274-a300-01a88aeb8458 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:07.027971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a639e2cf-4e77-4274-a300-01a88aeb8458 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:07.028253 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a639e2cf-4e77-4274-a300-01a88aeb8458 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:07.028529 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a639e2cf-4e77-4274-a300-01a88aeb8458 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:07.051895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64406ca0-6d77-40e2-a4e8-8f429e0ea520 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:07.056550 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a639e2cf-4e77-4274-a300-01a88aeb8458 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:07.058067 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a639e2cf-4e77-4274-a300-01a88aeb8458 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:07.088855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-a639e2cf-4e77-4274-a300-01a88aeb8458 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:07.088855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:07.088855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:07.088855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:07.088855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:07.088855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:07.088855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:07.088855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:07.088855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:07.088855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:07.088855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:07.088855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:07.088855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:07.088855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:07.088855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:07.088855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:07.089869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:07.089869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:07.089869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:07.089869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:07.089869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:07.089869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:07.089869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:07.089869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:07.089869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:07.089869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:07.089869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:07.089869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:07.089869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:07.089869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:07.089869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:07.089869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:07.089869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:07.090761 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:07.090761 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:07.090761 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:07.090761 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 344/692] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 327 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:07.098793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a639e2cf-4e77-4274-a300-01a88aeb8458 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:07.140479 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75085185-7cd5-459f-8e6f-ca3b20e39932 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:07.141995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75085185-7cd5-459f-8e6f-ca3b20e39932 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:07.142782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75085185-7cd5-459f-8e6f-ca3b20e39932 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:07.143911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75085185-7cd5-459f-8e6f-ca3b20e39932 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:07.160041 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75085185-7cd5-459f-8e6f-ca3b20e39932 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:07.161487 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75085185-7cd5-459f-8e6f-ca3b20e39932 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:07.184925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-75085185-7cd5-459f-8e6f-ca3b20e39932 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:07.184925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:07.184925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:07.184925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:07.184925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:07.184925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:07.184925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:07.184925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:07.184925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:07.184925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:07.184925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:07.184925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:07.184925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:07.184925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:07.184925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:07.185890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:07.185890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:07.185890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:07.185890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:07.185890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:07.185890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:07.185890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:07.185890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:07.185890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:07.185890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:07.185890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:07.185890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:07.185890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:07.185890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:07.185890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:07.185890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:07.185890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:07.186775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:07.186775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:07.186775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:07.186775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:07.186775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:07.186775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:07.186775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:07.186775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:07.186775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 349/693] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:07.190907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb260880-9c2d-48c3-863d-15e9d5583c6c None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:07.190907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb260880-9c2d-48c3-863d-15e9d5583c6c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:07.191223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb260880-9c2d-48c3-863d-15e9d5583c6c None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:07.192622 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 350/694] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:07.194324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb260880-9c2d-48c3-863d-15e9d5583c6c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:07.211292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:07.259096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a639e2cf-4e77-4274-a300-01a88aeb8458 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:07.296015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218d09e7-6ff7-4636-bc15-6b450bcc4582 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:07.297009 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218d09e7-6ff7-4636-bc15-6b450bcc4582 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:07.297490 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218d09e7-6ff7-4636-bc15-6b450bcc4582 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:07.297990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218d09e7-6ff7-4636-bc15-6b450bcc4582 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:07.298872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218d09e7-6ff7-4636-bc15-6b450bcc4582 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:07.299391 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218d09e7-6ff7-4636-bc15-6b450bcc4582 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:07.300942 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218d09e7-6ff7-4636-bc15-6b450bcc4582 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:07.314721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 345/695] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:07.320649 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb260880-9c2d-48c3-863d-15e9d5583c6c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:07.320933 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a36528-8d6f-4418-b956-6f67b7627839 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:07.321407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a36528-8d6f-4418-b956-6f67b7627839 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:07.321871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a36528-8d6f-4418-b956-6f67b7627839 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:07.341320 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c6a36528-8d6f-4418-b956-6f67b7627839 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:07.342846 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c6a36528-8d6f-4418-b956-6f67b7627839 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:07.352963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:07.359673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f9c41d-1cc9-4473-a247-584163dfdb99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:07.360368 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f9c41d-1cc9-4473-a247-584163dfdb99 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:07.360718 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f9c41d-1cc9-4473-a247-584163dfdb99 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:07.361072 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f9c41d-1cc9-4473-a247-584163dfdb99 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:07.361817 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f9c41d-1cc9-4473-a247-584163dfdb99 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:07.362190 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f9c41d-1cc9-4473-a247-584163dfdb99 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:07.363720 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f9c41d-1cc9-4473-a247-584163dfdb99 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:07.382496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 351/696] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:07.382496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f9c41d-1cc9-4473-a247-584163dfdb99 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:07.451365 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 346/697] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:06:07.451365 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a36528-8d6f-4418-b956-6f67b7627839 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:07.564307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:07.577241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f9c41d-1cc9-4473-a247-584163dfdb99 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:07.641575 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef7af4a-b7d2-4dd5-9849-f9b8ed652dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:07.642213 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef7af4a-b7d2-4dd5-9849-f9b8ed652dfd None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:07.642492 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef7af4a-b7d2-4dd5-9849-f9b8ed652dfd None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:07.642830 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef7af4a-b7d2-4dd5-9849-f9b8ed652dfd None admin] PATH_INFO: `/v3/users/nova` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:07.651253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef7af4a-b7d2-4dd5-9849-f9b8ed652dfd None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:07.652939 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef7af4a-b7d2-4dd5-9849-f9b8ed652dfd None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:07.666295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-8ef7af4a-b7d2-4dd5-9849-f9b8ed652dfd None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 01:06:07.666295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:07.666295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:07.666295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:07.666295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:07.666295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:07.666295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:07.666295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:07.666295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:07.666295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:07.666295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:07.666295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:07.666295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:07.666295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:07.666295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:07.666295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:07.667314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:07.667314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:07.667314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:07.667314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:07.667314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:07.667314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:07.667314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:07.667314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:07.667314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:07.667314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:07.667314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:07.667314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:07.667314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:07.667314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:07.667314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:07.667314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:07.667314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:07.667314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:07.668667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:07.668667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:07.668667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:07.668667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:07.668667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:07.668667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:07.668667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:07.668667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:07.668667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 01:06:07.668667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:07.668667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 352/698] 10.222.0.15 () {56 vars in 1093 bytes} [Tue Sep 13 01:06:07 2022] GET /identity/v3/users/nova => generated 82 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:07.673272 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef7af4a-b7d2-4dd5-9849-f9b8ed652dfd None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:07.674430 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a36528-8d6f-4418-b956-6f67b7627839 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:07.734889 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:07.748429 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0269b264-2412-4196-8455-9009bceb736e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:07.749134 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0269b264-2412-4196-8455-9009bceb736e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:07.749493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0269b264-2412-4196-8455-9009bceb736e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:07.749870 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0269b264-2412-4196-8455-9009bceb736e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:07.750633 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0269b264-2412-4196-8455-9009bceb736e None admin] RBAC: Adding query filter params (role.id=71b1e7c4b5af44ddbb7a517760ad0119, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=e0d7d8ed4ede4ddca192fce0a497689d) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:07.751027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0269b264-2412-4196-8455-9009bceb736e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:07.752521 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0269b264-2412-4196-8455-9009bceb736e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:07.769377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 347/699] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:06:07 2022] GET /identity/v3/role_assignments?user.id=e0d7d8ed4ede4ddca192fce0a497689d&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=71b1e7c4b5af44ddbb7a517760ad0119 => generated 608 bytes in 322 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:07.794083 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0269b264-2412-4196-8455-9009bceb736e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:07.805904 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef7af4a-b7d2-4dd5-9849-f9b8ed652dfd None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:07.848570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134621f1-273b-47aa-b8f8-4ff7e5b62cb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:07.849478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134621f1-273b-47aa-b8f8-4ff7e5b62cb8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:07.849938 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134621f1-273b-47aa-b8f8-4ff7e5b62cb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:07.851060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134621f1-273b-47aa-b8f8-4ff7e5b62cb8 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:07.851992 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134621f1-273b-47aa-b8f8-4ff7e5b62cb8 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:07.852469 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134621f1-273b-47aa-b8f8-4ff7e5b62cb8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:07.854086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134621f1-273b-47aa-b8f8-4ff7e5b62cb8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:07.877571 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 353/700] 10.222.0.15 () {56 vars in 1133 bytes} [Tue Sep 13 01:06:07 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:07.884499 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134621f1-273b-47aa-b8f8-4ff7e5b62cb8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:08.008212 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134621f1-273b-47aa-b8f8-4ff7e5b62cb8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:08.024000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:08.032404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0269b264-2412-4196-8455-9009bceb736e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:08.060365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839c6b70-6b4d-4070-b0e1-b0c6819f2e85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:08.061356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839c6b70-6b4d-4070-b0e1-b0c6819f2e85 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:08.062145 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839c6b70-6b4d-4070-b0e1-b0c6819f2e85 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:08.062657 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839c6b70-6b4d-4070-b0e1-b0c6819f2e85 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:08.076053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:08.099189 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca3d52f-4c87-46cf-9e6e-5b966fcc0bf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:08.100255 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca3d52f-4c87-46cf-9e6e-5b966fcc0bf7 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:08.100810 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca3d52f-4c87-46cf-9e6e-5b966fcc0bf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:08.101285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca3d52f-4c87-46cf-9e6e-5b966fcc0bf7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:08.117796 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca3d52f-4c87-46cf-9e6e-5b966fcc0bf7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:08.119658 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca3d52f-4c87-46cf-9e6e-5b966fcc0bf7 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:08.141274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-dca3d52f-4c87-46cf-9e6e-5b966fcc0bf7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:08.141274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:08.141274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:08.141274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:08.141274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:08.141274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:08.141274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:08.141274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:08.141274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:08.141274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:08.141274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:08.141274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:08.141274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:08.141274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:08.141274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:08.141274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:08.142281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:08.142281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:08.142281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:08.142281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:08.142281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:08.142281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:08.142281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:08.142281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:08.142281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:08.142281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:08.142281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:08.142281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:08.142281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:08.142281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:08.142281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:08.142281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:08.142281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:08.143315 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:08.143315 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:08.143315 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:08.145493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 348/701] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 352 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:08.148373 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839c6b70-6b4d-4070-b0e1-b0c6819f2e85 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:08.149858 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839c6b70-6b4d-4070-b0e1-b0c6819f2e85 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:08.151696 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca3d52f-4c87-46cf-9e6e-5b966fcc0bf7 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:08.266394 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca3d52f-4c87-46cf-9e6e-5b966fcc0bf7 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:08.278960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 354/702] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:06:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 01:06:08.279580 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03a2e91-c889-4416-a690-e8a7f5fec536 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:08.279868 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03a2e91-c889-4416-a690-e8a7f5fec536 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:08.280147 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03a2e91-c889-4416-a690-e8a7f5fec536 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:08.281706 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 355/703] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:08.284570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03a2e91-c889-4416-a690-e8a7f5fec536 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:08.299153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:08.314252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d320132-3c8b-4afa-9a51-c268f67cc737 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:08.315290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d320132-3c8b-4afa-9a51-c268f67cc737 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:08.315800 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d320132-3c8b-4afa-9a51-c268f67cc737 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:08.316272 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d320132-3c8b-4afa-9a51-c268f67cc737 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:08.317181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d320132-3c8b-4afa-9a51-c268f67cc737 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:08.317754 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d320132-3c8b-4afa-9a51-c268f67cc737 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:08.320030 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d320132-3c8b-4afa-9a51-c268f67cc737 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:08.335668 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 349/704] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:08.337282 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767d93f4-ce42-4b06-ba42-c32263210796 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:08.337771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767d93f4-ce42-4b06-ba42-c32263210796 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:08.338373 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767d93f4-ce42-4b06-ba42-c32263210796 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:08.364713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-767d93f4-ce42-4b06-ba42-c32263210796 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:08.365812 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-767d93f4-ce42-4b06-ba42-c32263210796 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:08.393082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:08.414347 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03a2e91-c889-4416-a690-e8a7f5fec536 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:08.448805 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 350/705] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:06:08.450016 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767d93f4-ce42-4b06-ba42-c32263210796 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:08.455441 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44b92c0-183e-4b46-894a-52efe0d590d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:08.456229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44b92c0-183e-4b46-894a-52efe0d590d8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:08.456751 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44b92c0-183e-4b46-894a-52efe0d590d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:08.457130 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44b92c0-183e-4b46-894a-52efe0d590d8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:08.471734 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44b92c0-183e-4b46-894a-52efe0d590d8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:08.473494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44b92c0-183e-4b46-894a-52efe0d590d8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:08.492286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-c44b92c0-183e-4b46-894a-52efe0d590d8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:08.492286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:08.492286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:08.492286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:08.492286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:08.492286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:08.492286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:08.492286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:08.492286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:08.492286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:08.492286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:08.492286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:08.492286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:08.492286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:08.492286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:08.492286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:08.493279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:08.493279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:08.493279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:08.493279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:08.493279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:08.493279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:08.493279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:08.493279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:08.493279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:08.493279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:08.493279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:08.493279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:08.493279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:08.493279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:08.493279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:08.493279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:08.493279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:08.494165 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:08.494165 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:08.494165 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:08.494165 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 356/706] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:08.495575 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d65831-5941-4fd4-8e61-6121233855b3 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:08.496804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d65831-5941-4fd4-8e61-6121233855b3 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:08.497151 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d65831-5941-4fd4-8e61-6121233855b3 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:08.498974 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 357/707] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:08 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:08.499754 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d65831-5941-4fd4-8e61-6121233855b3 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:08.575472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:08.590819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767d93f4-ce42-4b06-ba42-c32263210796 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:08.623754 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d65831-5941-4fd4-8e61-6121233855b3 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:08.634884 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64dcae1-8fae-4c5d-9a9b-6724fddc8161 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:08.634884 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64dcae1-8fae-4c5d-9a9b-6724fddc8161 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:08.634884 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64dcae1-8fae-4c5d-9a9b-6724fddc8161 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:08.634884 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64dcae1-8fae-4c5d-9a9b-6724fddc8161 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:08.635276 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f64dcae1-8fae-4c5d-9a9b-6724fddc8161 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:08.635276 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f64dcae1-8fae-4c5d-9a9b-6724fddc8161 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:08.662089 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 351/708] 10.222.0.15 () {60 vars in 1132 bytes} [Tue Sep 13 01:06:08 2022] POST /identity/v3/users => generated 254 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:06:08.663931 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5105c5-0a51-4e0a-a78f-47d1f6bbd8cf None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:08.664275 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5105c5-0a51-4e0a-a78f-47d1f6bbd8cf None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:08.664610 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5105c5-0a51-4e0a-a78f-47d1f6bbd8cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:08.677969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deaf61af-438d-4e81-8356-480c45eda6e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:08.679205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaf61af-438d-4e81-8356-480c45eda6e7 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:08.679488 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaf61af-438d-4e81-8356-480c45eda6e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:08.679762 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaf61af-438d-4e81-8356-480c45eda6e7 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:08.680418 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deaf61af-438d-4e81-8356-480c45eda6e7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:08.680720 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deaf61af-438d-4e81-8356-480c45eda6e7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:08.682139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deaf61af-438d-4e81-8356-480c45eda6e7 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:08.694604 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-cb5105c5-0a51-4e0a-a78f-47d1f6bbd8cf None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:08.696297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cb5105c5-0a51-4e0a-a78f-47d1f6bbd8cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:08.719209 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 358/709] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:08.720466 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deaf61af-438d-4e81-8356-480c45eda6e7 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:08.731264 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:08.797742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 352/710] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:06:08.798884 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5105c5-0a51-4e0a-a78f-47d1f6bbd8cf None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:08.855427 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deaf61af-438d-4e81-8356-480c45eda6e7 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:08.891342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:08.902334 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d62c7df-d0e2-429b-86ea-9f9bfc58450c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:08.903283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d62c7df-d0e2-429b-86ea-9f9bfc58450c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:08.903567 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d62c7df-d0e2-429b-86ea-9f9bfc58450c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:08.903841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d62c7df-d0e2-429b-86ea-9f9bfc58450c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:08.913910 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d62c7df-d0e2-429b-86ea-9f9bfc58450c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:08.915380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d62c7df-d0e2-429b-86ea-9f9bfc58450c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:08.930783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-5d62c7df-d0e2-429b-86ea-9f9bfc58450c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:08.930783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:08.930783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:08.930783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:08.930783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:08.930783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:08.930783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:08.930783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:08.930783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:08.930783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:08.930783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:08.930783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:08.930783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:08.930783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:08.930783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:08.931715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:08.931715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:08.931715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:08.931715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:08.931715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:08.931715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:08.931715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:08.931715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:08.931715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:08.931715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:08.931715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:08.931715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:08.931715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:08.931715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:08.931715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:08.931715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:08.931715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:08.932598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:08.932598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:08.932598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:08.932598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:08.932598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:08.932598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:08.932598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:08.932598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:08.932598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 359/711] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:08.938914 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d62c7df-d0e2-429b-86ea-9f9bfc58450c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:08.958089 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5105c5-0a51-4e0a-a78f-47d1f6bbd8cf None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:09.011012 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc4b0ce-7327-4144-afd6-9b1abde248a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:09.012249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc4b0ce-7327-4144-afd6-9b1abde248a5 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:09.013151 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc4b0ce-7327-4144-afd6-9b1abde248a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:09.013899 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc4b0ce-7327-4144-afd6-9b1abde248a5 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:09.014911 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc4b0ce-7327-4144-afd6-9b1abde248a5 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:09.017169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc4b0ce-7327-4144-afd6-9b1abde248a5 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:09.031886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-0cc4b0ce-7327-4144-afd6-9b1abde248a5 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 01:06:09.031886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:09.031886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:09.031886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:09.031886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:09.031886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:09.031886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:09.031886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:09.031886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 01:06:09.031886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:09.031886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:09.031886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:09.031886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:09.031886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:09.031886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:09.031886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:09.031886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:09.033145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:09.033145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:09.033145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:09.033145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:09.033145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:09.033145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:09.033145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:09.033145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:09.033145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:09.033145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:09.033145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:09.033145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:09.033145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:09.033145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:09.033145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:09.033145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:09.034173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:09.034173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:09.034173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:09.034173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:09.034173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:09.034173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:09.034173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:09.034173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:09.034173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:09.034173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:09.034173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:09.034173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:09.034173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 01:06:09.034173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:09.036739 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 353/712] 10.222.0.15 () {56 vars in 1129 bytes} [Tue Sep 13 01:06:08 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:09.044551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc4b0ce-7327-4144-afd6-9b1abde248a5 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:09.060135 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:09.071110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d62c7df-d0e2-429b-86ea-9f9bfc58450c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:09.110510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5e14e6-d0f7-451c-9e05-7f3ebd6f4949 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:09.111269 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5e14e6-d0f7-451c-9e05-7f3ebd6f4949 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:09.111551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5e14e6-d0f7-451c-9e05-7f3ebd6f4949 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:09.111824 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5e14e6-d0f7-451c-9e05-7f3ebd6f4949 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:09.112512 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5e14e6-d0f7-451c-9e05-7f3ebd6f4949 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:09.112818 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5e14e6-d0f7-451c-9e05-7f3ebd6f4949 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:09.114364 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5e14e6-d0f7-451c-9e05-7f3ebd6f4949 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:09.132036 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 360/713] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:09.133642 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5e14e6-d0f7-451c-9e05-7f3ebd6f4949 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:09.183095 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc4b0ce-7327-4144-afd6-9b1abde248a5 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:09.193027 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:09.219899 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd702658-178b-4485-acc7-ef058cb691e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:09.220509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd702658-178b-4485-acc7-ef058cb691e8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:09.220787 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd702658-178b-4485-acc7-ef058cb691e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:09.221057 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd702658-178b-4485-acc7-ef058cb691e8 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:09.221677 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd702658-178b-4485-acc7-ef058cb691e8 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:09.221977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd702658-178b-4485-acc7-ef058cb691e8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:09.223388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd702658-178b-4485-acc7-ef058cb691e8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:09.228348 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 354/714] 10.222.0.15 () {56 vars in 1124 bytes} [Tue Sep 13 01:06:09 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:09.229135 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd702658-178b-4485-acc7-ef058cb691e8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:09.274394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5e14e6-d0f7-451c-9e05-7f3ebd6f4949 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:09.316309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca288be-02eb-4e8a-9ed0-8e0d8c9a1d9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:09.316942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca288be-02eb-4e8a-9ed0-8e0d8c9a1d9d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:09.317218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca288be-02eb-4e8a-9ed0-8e0d8c9a1d9d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:09.317488 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca288be-02eb-4e8a-9ed0-8e0d8c9a1d9d None admin] PATH_INFO: `/v3/services/network` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:09.318061 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca288be-02eb-4e8a-9ed0-8e0d8c9a1d9d None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:09.318567 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:09.320617 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca288be-02eb-4e8a-9ed0-8e0d8c9a1d9d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:09.335024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-8ca288be-02eb-4e8a-9ed0-8e0d8c9a1d9d None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 01:06:09.335024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:09.335024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:09.335024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:09.335024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:09.335024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:09.335024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:09.335024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:09.335024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 01:06:09.335024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:09.335024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:09.335024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:09.335024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:09.335024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:09.335024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:09.335024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:09.335024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:09.337374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:09.337374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:09.337374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:09.337374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:09.337374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:09.337374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:09.337374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:09.337374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:09.337374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:09.337374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:09.337374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:09.337374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:09.337374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:09.337374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:09.337374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:09.337374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:09.339401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:09.339401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:09.339401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:09.339401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:09.339401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:09.339401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:09.339401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:09.339401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:09.339401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:09.339401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:09.339401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:09.339401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:09.339401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 01:06:09.339401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:09.339401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 361/715] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:09 2022] GET /identity/v3/services/network => generated 88 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:09.341067 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca288be-02eb-4e8a-9ed0-8e0d8c9a1d9d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:09.358882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd702658-178b-4485-acc7-ef058cb691e8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:09.398105 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b38d41-d805-4ae8-89fc-42fcec6dddc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:09.399008 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b38d41-d805-4ae8-89fc-42fcec6dddc0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:09.399286 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b38d41-d805-4ae8-89fc-42fcec6dddc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:09.399807 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b38d41-d805-4ae8-89fc-42fcec6dddc0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:09.400909 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b38d41-d805-4ae8-89fc-42fcec6dddc0 None admin] RBAC: Adding query filter params (role.id=66a36c858aab40688080eed6ceace3a0, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=2f2da4d2d3604d16b58dc25b8c17c49a) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:09.401218 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b38d41-d805-4ae8-89fc-42fcec6dddc0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:09.402775 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b38d41-d805-4ae8-89fc-42fcec6dddc0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:09.422903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 355/716] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:06:09 2022] GET /identity/v3/role_assignments?user.id=2f2da4d2d3604d16b58dc25b8c17c49a&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=66a36c858aab40688080eed6ceace3a0 => generated 259 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:09.422903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b38d41-d805-4ae8-89fc-42fcec6dddc0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:09.520518 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca288be-02eb-4e8a-9ed0-8e0d8c9a1d9d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:09.533999 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:09.572391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d73ac1-aa67-4beb-9997-d34c6fc15f55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:09.573501 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d73ac1-aa67-4beb-9997-d34c6fc15f55 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:09.573993 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d73ac1-aa67-4beb-9997-d34c6fc15f55 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:09.574467 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d73ac1-aa67-4beb-9997-d34c6fc15f55 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:09.575325 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d73ac1-aa67-4beb-9997-d34c6fc15f55 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:09.575850 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d73ac1-aa67-4beb-9997-d34c6fc15f55 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:09.577694 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d73ac1-aa67-4beb-9997-d34c6fc15f55 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:09.589926 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 362/717] 10.222.0.15 () {56 vars in 1124 bytes} [Tue Sep 13 01:06:09 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:09.642508 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b38d41-d805-4ae8-89fc-42fcec6dddc0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:09.689968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6d66a7-4983-47f3-833a-7a5dfe581141 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:09.690596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6d66a7-4983-47f3-833a-7a5dfe581141 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:09.690886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6d66a7-4983-47f3-833a-7a5dfe581141 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:09.691158 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6d66a7-4983-47f3-833a-7a5dfe581141 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:09.693159 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6d66a7-4983-47f3-833a-7a5dfe581141 None admin] RBAC: Adding query filter params (name=network) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:09.693159 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6d66a7-4983-47f3-833a-7a5dfe581141 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:09.693428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6d66a7-4983-47f3-833a-7a5dfe581141 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:09.699385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 356/718] 10.222.0.15 () {56 vars in 1112 bytes} [Tue Sep 13 01:06:09 2022] GET /identity/v3/services?name=network => generated 124 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:09.704743 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d73ac1-aa67-4beb-9997-d34c6fc15f55 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:09.833597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d73ac1-aa67-4beb-9997-d34c6fc15f55 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:09.878249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b9dbb8-70f4-4ce5-9011-8b249ec81216 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:09.879291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b9dbb8-70f4-4ce5-9011-8b249ec81216 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:09.879778 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b9dbb8-70f4-4ce5-9011-8b249ec81216 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:09.880255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b9dbb8-70f4-4ce5-9011-8b249ec81216 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:09.881129 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b9dbb8-70f4-4ce5-9011-8b249ec81216 None admin] RBAC: Adding query filter params (type=network) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:09.881706 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b9dbb8-70f4-4ce5-9011-8b249ec81216 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:09.883368 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b9dbb8-70f4-4ce5-9011-8b249ec81216 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:09.890410 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 363/719] 10.222.0.15 () {56 vars in 1112 bytes} [Tue Sep 13 01:06:09 2022] GET /identity/v3/services?type=network => generated 124 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:10.540874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fad936e-4586-40a1-996b-0ba04133bb26 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:10.541427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fad936e-4586-40a1-996b-0ba04133bb26 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:10.541427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fad936e-4586-40a1-996b-0ba04133bb26 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:10.543113 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 357/720] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:10.548549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1284fc56-7888-4993-b86a-01cd667ead0d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:10.549062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1284fc56-7888-4993-b86a-01cd667ead0d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:10.549221 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1284fc56-7888-4993-b86a-01cd667ead0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:10.576010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-1284fc56-7888-4993-b86a-01cd667ead0d None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:10.577248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1284fc56-7888-4993-b86a-01cd667ead0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:10.609134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:10.669406 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 364/721] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:06:11.057144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fad936e-4586-40a1-996b-0ba04133bb26 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:11.060095 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:11.203699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fad936e-4586-40a1-996b-0ba04133bb26 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:11.247110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1711e1-e952-46b9-abb0-c201c621681d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:11.247736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1711e1-e952-46b9-abb0-c201c621681d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:11.248015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1711e1-e952-46b9-abb0-c201c621681d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:11.248287 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1711e1-e952-46b9-abb0-c201c621681d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:11.263958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1711e1-e952-46b9-abb0-c201c621681d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:11.269340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-6b1711e1-e952-46b9-abb0-c201c621681d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:06:11.269340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:11.269340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:11.269340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:11.269340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:11.269340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:11.269340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:11.269340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:11.269340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:11.269340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:11.269340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:11.269340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:11.269340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:11.269340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:11.269340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:11.269340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:11.270244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:11.270244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:11.270244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:11.270244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:11.270244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:11.270244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:11.270244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:11.270244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:11.270244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:11.270244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:11.270244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:11.270244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:11.270244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:11.270244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:11.270244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:11.270244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:11.271197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:11.271197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:11.271197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:11.271197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:11.271197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:11.271197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:11.271197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:11.271197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:11.271197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:11.271197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:11.271197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:06:11.271197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:11.273768 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 358/722] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:06:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:11.283935 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1284fc56-7888-4993-b86a-01cd667ead0d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:11.444676 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1284fc56-7888-4993-b86a-01cd667ead0d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:11.488737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699f22b7-e35f-46a1-bb12-398a4e973b18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:11.489355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699f22b7-e35f-46a1-bb12-398a4e973b18 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:11.489628 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699f22b7-e35f-46a1-bb12-398a4e973b18 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:11.489896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699f22b7-e35f-46a1-bb12-398a4e973b18 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:11.490553 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699f22b7-e35f-46a1-bb12-398a4e973b18 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:11.490954 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699f22b7-e35f-46a1-bb12-398a4e973b18 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:11.492303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699f22b7-e35f-46a1-bb12-398a4e973b18 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:11.507205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 365/723] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:06:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:11.512847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1711e1-e952-46b9-abb0-c201c621681d None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:11.529899 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc653ec-535f-4740-bd04-055cf76b374c None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:11.530156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc653ec-535f-4740-bd04-055cf76b374c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:11.530433 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc653ec-535f-4740-bd04-055cf76b374c None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:11.532230 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 366/724] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:06:11.537596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b088cdba-b317-42b4-b58f-303436dbcc0a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:11.538416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b088cdba-b317-42b4-b58f-303436dbcc0a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:11.538678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b088cdba-b317-42b4-b58f-303436dbcc0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:11.577816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b088cdba-b317-42b4-b58f-303436dbcc0a None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:11.579019 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b088cdba-b317-42b4-b58f-303436dbcc0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:11.603388 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:11.642557 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:11.701988 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 367/725] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:06:11.715476 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1711e1-e952-46b9-abb0-c201c621681d None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:11.760669 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3aa26b-ace4-469f-a9de-74a71457e39c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:11.761303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3aa26b-ace4-469f-a9de-74a71457e39c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:11.761580 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3aa26b-ace4-469f-a9de-74a71457e39c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:11.761849 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3aa26b-ace4-469f-a9de-74a71457e39c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:11.778837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3aa26b-ace4-469f-a9de-74a71457e39c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:11.780289 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3aa26b-ace4-469f-a9de-74a71457e39c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:11.801130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-9a3aa26b-ace4-469f-a9de-74a71457e39c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:11.801130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:11.801130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:11.801130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:11.801130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:11.801130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:11.801130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:11.801130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:11.801130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:11.801130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:11.801130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:11.801130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:11.801130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:11.801130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:11.801130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:11.801130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:11.802106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:11.802106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:11.802106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:11.802106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:11.802106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:11.802106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:11.802106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:11.802106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:11.802106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:11.802106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:11.802106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:11.802106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:11.802106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:11.802106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:11.802106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:11.802106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:11.802106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:11.803051 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:11.803051 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:11.803051 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:11.803051 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 359/726] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:11.809804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b088cdba-b317-42b4-b58f-303436dbcc0a None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:11.954789 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b088cdba-b317-42b4-b58f-303436dbcc0a None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:12.008133 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9aa1ba1-3593-4d6c-95d6-241cb7e6cbee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:12.008762 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9aa1ba1-3593-4d6c-95d6-241cb7e6cbee None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:12.009044 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9aa1ba1-3593-4d6c-95d6-241cb7e6cbee None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:12.009320 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9aa1ba1-3593-4d6c-95d6-241cb7e6cbee None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:12.009964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9aa1ba1-3593-4d6c-95d6-241cb7e6cbee None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:12.010269 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9aa1ba1-3593-4d6c-95d6-241cb7e6cbee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:12.011654 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9aa1ba1-3593-4d6c-95d6-241cb7e6cbee None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:12.029482 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118ddfa0-4b34-412e-9341-34160981a683 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:12.029787 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118ddfa0-4b34-412e-9341-34160981a683 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:12.030024 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118ddfa0-4b34-412e-9341-34160981a683 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:12.031214 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 368/727] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:12.031599 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 360/728] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:12.033491 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9aa1ba1-3593-4d6c-95d6-241cb7e6cbee None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:12.036266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b233a88-5e47-49df-8cc3-8bdc25049fb4 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:12.036763 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b233a88-5e47-49df-8cc3-8bdc25049fb4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:12.037249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b233a88-5e47-49df-8cc3-8bdc25049fb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:12.066428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7b233a88-5e47-49df-8cc3-8bdc25049fb4 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:12.068177 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7b233a88-5e47-49df-8cc3-8bdc25049fb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:12.079195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:12.159005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 361/729] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:06:12.160184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b233a88-5e47-49df-8cc3-8bdc25049fb4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:12.187800 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9aa1ba1-3593-4d6c-95d6-241cb7e6cbee None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:12.227638 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20478748-d920-4d8e-b796-8965d77f233b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:12.228562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20478748-d920-4d8e-b796-8965d77f233b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:12.228842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20478748-d920-4d8e-b796-8965d77f233b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:12.229114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20478748-d920-4d8e-b796-8965d77f233b None admin] PATH_INFO: `/v3/users/swift` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:12.237791 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20478748-d920-4d8e-b796-8965d77f233b None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:12.239235 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20478748-d920-4d8e-b796-8965d77f233b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:12.252517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-20478748-d920-4d8e-b796-8965d77f233b None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 13 01:06:12.252517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:12.252517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:12.252517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:12.252517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:12.252517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:12.252517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:12.252517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:12.252517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:12.252517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:12.252517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:12.252517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:12.252517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:12.252517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:12.252517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:12.252517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:12.253456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:12.253456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:12.253456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:12.253456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:12.253456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:12.253456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:12.253456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:12.253456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:12.253456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:12.253456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:12.253456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:12.253456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:12.253456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:12.253456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:12.253456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:12.253456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:12.253456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:12.253456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:12.254404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:12.254404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:12.254404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:12.254404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:12.254404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:12.254404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:12.254404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:12.254404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:12.254404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 13 01:06:12.254404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:12.254404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 369/730] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:06:12 2022] GET /identity/v3/users/swift => generated 83 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:12.259863 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20478748-d920-4d8e-b796-8965d77f233b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:12.279043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:12.357893 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:12.384879 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20478748-d920-4d8e-b796-8965d77f233b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:12.426593 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf93b7d-5bdb-4fdc-a193-673fe730f84a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:12.427245 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf93b7d-5bdb-4fdc-a193-673fe730f84a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:12.427520 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf93b7d-5bdb-4fdc-a193-673fe730f84a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:12.427790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf93b7d-5bdb-4fdc-a193-673fe730f84a None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:12.428640 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf93b7d-5bdb-4fdc-a193-673fe730f84a None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:12.428946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf93b7d-5bdb-4fdc-a193-673fe730f84a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:12.430391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf93b7d-5bdb-4fdc-a193-673fe730f84a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:12.435260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b233a88-5e47-49df-8cc3-8bdc25049fb4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:12.451815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 370/731] 10.222.0.15 () {56 vars in 1135 bytes} [Tue Sep 13 01:06:12 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:12.458595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf93b7d-5bdb-4fdc-a193-673fe730f84a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:12.489706 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753ad710-07c2-4969-93d7-fd0f37c1e418 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:12.490354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753ad710-07c2-4969-93d7-fd0f37c1e418 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:12.490639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753ad710-07c2-4969-93d7-fd0f37c1e418 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:12.491847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753ad710-07c2-4969-93d7-fd0f37c1e418 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:12.504260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753ad710-07c2-4969-93d7-fd0f37c1e418 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:12.509771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-753ad710-07c2-4969-93d7-fd0f37c1e418 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:06:12.509771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:12.509771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:12.509771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:12.509771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:12.509771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:12.509771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:12.509771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:12.509771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:12.509771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:12.509771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:12.509771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:12.509771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:12.509771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:12.509771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:12.509771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:12.510808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:12.510808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:12.510808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:12.510808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:12.510808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:12.510808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:12.510808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:12.510808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:12.510808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:12.510808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:12.510808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:12.510808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:12.510808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:12.510808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:12.510808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:12.510808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:12.511689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:12.511689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:12.511689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:12.511689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:12.511689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:12.511689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:12.511689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:12.511689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:12.511689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:12.511689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:12.511689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:06:12.511689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:12.516269 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 362/732] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:06:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 352 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:12.517296 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753ad710-07c2-4969-93d7-fd0f37c1e418 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:12.574291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf93b7d-5bdb-4fdc-a193-673fe730f84a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:12.611507 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e114e2fd-3073-4687-b4d3-c652a19e524e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:12.612383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e114e2fd-3073-4687-b4d3-c652a19e524e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:12.613063 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e114e2fd-3073-4687-b4d3-c652a19e524e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:12.613348 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e114e2fd-3073-4687-b4d3-c652a19e524e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:12.617056 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:12.629612 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753ad710-07c2-4969-93d7-fd0f37c1e418 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:12.632847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:12.665840 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c09af6-f92f-4c6f-bff8-40a8707c0538 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:12.666467 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c09af6-f92f-4c6f-bff8-40a8707c0538 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:12.666747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c09af6-f92f-4c6f-bff8-40a8707c0538 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:12.667050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c09af6-f92f-4c6f-bff8-40a8707c0538 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:12.677097 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c09af6-f92f-4c6f-bff8-40a8707c0538 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:12.681976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-d4c09af6-f92f-4c6f-bff8-40a8707c0538 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:06:12.681976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:12.681976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:12.681976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:12.681976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:12.681976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:12.681976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:12.681976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:12.681976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:12.681976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:12.681976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:12.681976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:12.681976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:12.681976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:12.681976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:12.681976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:12.682968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:12.682968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:12.682968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:12.682968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:12.682968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:12.682968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:12.682968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:12.682968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:12.682968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:12.682968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:12.682968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:12.682968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:12.682968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:12.682968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:12.682968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:12.682968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:12.683832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:12.683832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:12.683832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:12.683832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:12.683832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:12.683832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:12.683832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:12.683832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:12.683832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:12.683832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:12.683832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:06:12.683832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:12.683832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 363/733] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:06:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:12.683832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c09af6-f92f-4c6f-bff8-40a8707c0538 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:12.693470 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e114e2fd-3073-4687-b4d3-c652a19e524e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:12.695031 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e114e2fd-3073-4687-b4d3-c652a19e524e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:12.785933 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c09af6-f92f-4c6f-bff8-40a8707c0538 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:12.791803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 371/734] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:06:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 13 01:06:12.793186 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ff7b17-5f39-4abe-a090-30717a7c89f3 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:12.793477 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ff7b17-5f39-4abe-a090-30717a7c89f3 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:12.793750 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ff7b17-5f39-4abe-a090-30717a7c89f3 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:12.798199 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 372/735] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:12.798199 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395cd048-1611-4784-bd85-b9a7cc0afe23 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:12.798199 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395cd048-1611-4784-bd85-b9a7cc0afe23 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:12.798199 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395cd048-1611-4784-bd85-b9a7cc0afe23 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:12.798199 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 373/736] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:12 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:12.799054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395cd048-1611-4784-bd85-b9a7cc0afe23 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:12.827450 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4f9a43-8707-48e6-ab70-1f7d0f129f73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:12.828061 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4f9a43-8707-48e6-ab70-1f7d0f129f73 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:12.828337 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4f9a43-8707-48e6-ab70-1f7d0f129f73 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:12.828608 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4f9a43-8707-48e6-ab70-1f7d0f129f73 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:12.829263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4f9a43-8707-48e6-ab70-1f7d0f129f73 None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:12.829563 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4f9a43-8707-48e6-ab70-1f7d0f129f73 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:12.830968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4f9a43-8707-48e6-ab70-1f7d0f129f73 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:12.835132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:12.839346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 364/737] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:06:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:12.840146 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4f9a43-8707-48e6-ab70-1f7d0f129f73 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:12.866801 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:12.903300 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395cd048-1611-4784-bd85-b9a7cc0afe23 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:12.935926 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71941be-1405-4fb2-9108-b9a017a64a41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:12.936519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71941be-1405-4fb2-9108-b9a017a64a41 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:12.936793 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71941be-1405-4fb2-9108-b9a017a64a41 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:12.937061 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71941be-1405-4fb2-9108-b9a017a64a41 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:12.937398 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4f9a43-8707-48e6-ab70-1f7d0f129f73 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:12.937704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71941be-1405-4fb2-9108-b9a017a64a41 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:12.938267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71941be-1405-4fb2-9108-b9a017a64a41 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:12.939632 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71941be-1405-4fb2-9108-b9a017a64a41 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:12.947985 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 374/738] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:06:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:12.949120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbe2aad-2663-4c07-b496-1edc31dac26d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:12.949381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbe2aad-2663-4c07-b496-1edc31dac26d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:12.949650 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbe2aad-2663-4c07-b496-1edc31dac26d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:12.967366 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9fbe2aad-2663-4c07-b496-1edc31dac26d None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:12.968516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9fbe2aad-2663-4c07-b496-1edc31dac26d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:12.972240 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc7d177-f772-4d40-8504-026023dd61fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:12.972869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc7d177-f772-4d40-8504-026023dd61fa None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:12.973152 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc7d177-f772-4d40-8504-026023dd61fa None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:12.973435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc7d177-f772-4d40-8504-026023dd61fa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:12.983716 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc7d177-f772-4d40-8504-026023dd61fa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:12.985144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc7d177-f772-4d40-8504-026023dd61fa None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:12.998792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-1dc7d177-f772-4d40-8504-026023dd61fa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:12.998792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:12.998792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:12.998792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:12.998792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:12.998792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:12.998792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:12.998792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:12.998792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:12.998792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:12.998792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:12.998792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:12.998792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:12.998792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:12.998792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:12.998792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:12.999724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:12.999724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:12.999724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:12.999724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:12.999724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:12.999724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:12.999724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:12.999724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:12.999724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:12.999724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:12.999724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:12.999724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:12.999724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:12.999724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:12.999724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:12.999724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:12.999724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:13.000619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:13.000619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:13.000619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:13.000619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 365/739] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:13.001403 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118389b0-27d4-4e70-be2f-b411b8679408 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:13.001403 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118389b0-27d4-4e70-be2f-b411b8679408 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:13.001574 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118389b0-27d4-4e70-be2f-b411b8679408 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:13.019177 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-118389b0-27d4-4e70-be2f-b411b8679408 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:13.020354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-118389b0-27d4-4e70-be2f-b411b8679408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:13.051020 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 375/740] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:06:13.051020 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbe2aad-2663-4c07-b496-1edc31dac26d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:13.069417 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:13.090455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 366/741] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 01:06:13.091958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118389b0-27d4-4e70-be2f-b411b8679408 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:13.113402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:13.151087 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbe2aad-2663-4c07-b496-1edc31dac26d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:13.190668 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfca0f9b-103b-4c93-bc23-5402d6f04a91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:13.191346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfca0f9b-103b-4c93-bc23-5402d6f04a91 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:13.191628 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfca0f9b-103b-4c93-bc23-5402d6f04a91 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:13.191897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfca0f9b-103b-4c93-bc23-5402d6f04a91 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:13.203636 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfca0f9b-103b-4c93-bc23-5402d6f04a91 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:13.205121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfca0f9b-103b-4c93-bc23-5402d6f04a91 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:13.220897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-dfca0f9b-103b-4c93-bc23-5402d6f04a91 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:13.220897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:13.220897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:13.220897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:13.220897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:13.220897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:13.220897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:13.220897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:13.220897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:13.220897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:13.220897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:13.220897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:13.220897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:13.220897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:13.220897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:13.220897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:13.221853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:13.221853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:13.221853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:13.221853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:13.221853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:13.221853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:13.221853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:13.221853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:13.221853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:13.221853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:13.221853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:13.221853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:13.221853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:13.221853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:13.221853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:13.221853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:13.221853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:13.222733 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:13.222733 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:13.222733 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:13.226804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 376/742] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:13.227649 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfca0f9b-103b-4c93-bc23-5402d6f04a91 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:13.234782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118389b0-27d4-4e70-be2f-b411b8679408 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:13.281522 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd2a5cc-980f-4911-bd2b-acc0da1ca680 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:13.282523 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd2a5cc-980f-4911-bd2b-acc0da1ca680 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:13.283055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd2a5cc-980f-4911-bd2b-acc0da1ca680 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:13.283534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd2a5cc-980f-4911-bd2b-acc0da1ca680 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:13.298482 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd2a5cc-980f-4911-bd2b-acc0da1ca680 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:13.300481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd2a5cc-980f-4911-bd2b-acc0da1ca680 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:13.311924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:13.319627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-abd2a5cc-980f-4911-bd2b-acc0da1ca680 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:13.319627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:13.319627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:13.319627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:13.319627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:13.319627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:13.319627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:13.319627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:13.319627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:13.319627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:13.319627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:13.319627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:13.319627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:13.319627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:13.319627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:13.319627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:13.320750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:13.320750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:13.320750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:13.320750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:13.320750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:13.320750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:13.320750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:13.320750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:13.320750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:13.320750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:13.320750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:13.320750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:13.320750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:13.320750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:13.320750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:13.320750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:13.320750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:13.321642 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:13.321642 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:13.321642 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:13.321642 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 367/743] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:13.321642 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd2a5cc-980f-4911-bd2b-acc0da1ca680 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:13.340629 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfca0f9b-103b-4c93-bc23-5402d6f04a91 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:13.375900 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e0fc99-6371-4bd5-af2c-9639b4fd6617 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:13.377159 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e0fc99-6371-4bd5-af2c-9639b4fd6617 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:13.377728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e0fc99-6371-4bd5-af2c-9639b4fd6617 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:13.378204 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e0fc99-6371-4bd5-af2c-9639b4fd6617 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:13.379008 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e0fc99-6371-4bd5-af2c-9639b4fd6617 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:13.379315 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e0fc99-6371-4bd5-af2c-9639b4fd6617 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:13.381208 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e0fc99-6371-4bd5-af2c-9639b4fd6617 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:13.384008 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:13.394100 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 377/744] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:13.394900 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e0fc99-6371-4bd5-af2c-9639b4fd6617 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:13.432161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd2a5cc-980f-4911-bd2b-acc0da1ca680 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:13.465139 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060ef45c-d659-43f8-b087-0cae62cdaa93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:13.465742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060ef45c-d659-43f8-b087-0cae62cdaa93 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:13.466028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060ef45c-d659-43f8-b087-0cae62cdaa93 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:13.466301 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060ef45c-d659-43f8-b087-0cae62cdaa93 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:13.466967 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060ef45c-d659-43f8-b087-0cae62cdaa93 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:13.467269 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060ef45c-d659-43f8-b087-0cae62cdaa93 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:13.468627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060ef45c-d659-43f8-b087-0cae62cdaa93 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:13.480117 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 368/745] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:13.480896 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060ef45c-d659-43f8-b087-0cae62cdaa93 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:13.500816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e0fc99-6371-4bd5-af2c-9639b4fd6617 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:13.536264 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224d1139-feee-48fc-86c5-df493c4fe4d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:13.536866 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224d1139-feee-48fc-86c5-df493c4fe4d5 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:13.537143 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224d1139-feee-48fc-86c5-df493c4fe4d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:13.537419 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224d1139-feee-48fc-86c5-df493c4fe4d5 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:13.538110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224d1139-feee-48fc-86c5-df493c4fe4d5 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:13.539491 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224d1139-feee-48fc-86c5-df493c4fe4d5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:13.549385 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:13.556584 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 378/746] 10.222.0.15 () {60 vars in 1141 bytes} [Tue Sep 13 01:06:13 2022] POST /identity/v3/services => generated 251 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:06:13.557345 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224d1139-feee-48fc-86c5-df493c4fe4d5 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:13.586495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060ef45c-d659-43f8-b087-0cae62cdaa93 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:13.612132 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:13.637933 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf6d285-25a0-431e-a2be-c37311ce8d2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:13.638559 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf6d285-25a0-431e-a2be-c37311ce8d2e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:13.638841 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf6d285-25a0-431e-a2be-c37311ce8d2e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:13.639148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf6d285-25a0-431e-a2be-c37311ce8d2e None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:13.639806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf6d285-25a0-431e-a2be-c37311ce8d2e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:13.640110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf6d285-25a0-431e-a2be-c37311ce8d2e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:13.641506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf6d285-25a0-431e-a2be-c37311ce8d2e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:13.658965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 369/747] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:13.658965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf6d285-25a0-431e-a2be-c37311ce8d2e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:13.681432 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224d1139-feee-48fc-86c5-df493c4fe4d5 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:13.720570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336a2bf6-8777-4b8f-98fe-55c296761450 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:13.721311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336a2bf6-8777-4b8f-98fe-55c296761450 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:13.721594 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336a2bf6-8777-4b8f-98fe-55c296761450 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:13.721866 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336a2bf6-8777-4b8f-98fe-55c296761450 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:13.722518 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336a2bf6-8777-4b8f-98fe-55c296761450 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:13.723966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336a2bf6-8777-4b8f-98fe-55c296761450 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:13.740264 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 379/748] 10.222.0.15 () {60 vars in 1141 bytes} [Tue Sep 13 01:06:13 2022] POST /identity/v3/services => generated 240 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:06:13.741769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336a2bf6-8777-4b8f-98fe-55c296761450 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:13.771551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf6d285-25a0-431e-a2be-c37311ce8d2e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:13.822488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4dfd563-9dfe-441a-aba8-6ec976d62a41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:13.823168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dfd563-9dfe-441a-aba8-6ec976d62a41 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:13.823446 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dfd563-9dfe-441a-aba8-6ec976d62a41 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:13.823712 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dfd563-9dfe-441a-aba8-6ec976d62a41 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:13.823904 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:13.833598 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4dfd563-9dfe-441a-aba8-6ec976d62a41 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:13.834548 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4dfd563-9dfe-441a-aba8-6ec976d62a41 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:13.849449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-c4dfd563-9dfe-441a-aba8-6ec976d62a41 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:13.849449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:13.849449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:13.849449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:13.849449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:13.849449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:13.849449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:13.849449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:13.849449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:13.849449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:13.849449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:13.849449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:13.849449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:13.849449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:13.849449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:13.850974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:13.850974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:13.850974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:13.850974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:13.850974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:13.850974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:13.850974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:13.850974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:13.850974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:13.850974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:13.850974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:13.850974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:13.850974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:13.850974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:13.850974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:13.850974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:13.850974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:13.851871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:13.851871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:13.851871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:13.851871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:13.851871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:13.851871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:13.851871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:13.851871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:13.851871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 370/749] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:13.851871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4dfd563-9dfe-441a-aba8-6ec976d62a41 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:13.891487 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:13.898700 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336a2bf6-8777-4b8f-98fe-55c296761450 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:13.946215 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90beaba8-7fb6-4e98-87ea-f1306887a238 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:13.946917 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90beaba8-7fb6-4e98-87ea-f1306887a238 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:13.947214 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90beaba8-7fb6-4e98-87ea-f1306887a238 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:13.947495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90beaba8-7fb6-4e98-87ea-f1306887a238 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:13.957502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90beaba8-7fb6-4e98-87ea-f1306887a238 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:13.959153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90beaba8-7fb6-4e98-87ea-f1306887a238 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:13.979037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-90beaba8-7fb6-4e98-87ea-f1306887a238 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 01:06:13.979037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:13.979037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:13.979037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:13.979037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:13.979037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:13.979037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:13.979037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:13.979037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:13.979037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:13.979037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:13.979037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:13.979037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:13.979037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:13.979037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:13.979037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:13.980052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:13.980052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:13.980052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:13.980052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:13.980052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:13.980052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:13.980052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:13.980052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:13.980052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:13.980052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:13.980052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:13.980052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:13.980052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:13.980052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:13.980052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:13.980052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:13.980052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:13.980052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:13.981033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:13.981033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:13.981033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:13.981033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:13.981033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:13.981033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:13.981033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:13.981033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:13.981033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 01:06:13.981033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:13.981578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4dfd563-9dfe-441a-aba8-6ec976d62a41 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:13.981652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 380/750] 10.222.0.15 () {56 vars in 1117 bytes} [Tue Sep 13 01:06:13 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:13.982714 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90beaba8-7fb6-4e98-87ea-f1306887a238 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:14.024267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6e8518-0774-45d3-8e74-ae98d3ceb2f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:14.025177 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6e8518-0774-45d3-8e74-ae98d3ceb2f3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:14.025550 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6e8518-0774-45d3-8e74-ae98d3ceb2f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:14.025918 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6e8518-0774-45d3-8e74-ae98d3ceb2f3 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:14.035167 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6e8518-0774-45d3-8e74-ae98d3ceb2f3 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:14.036778 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6e8518-0774-45d3-8e74-ae98d3ceb2f3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:14.049923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-df6e8518-0774-45d3-8e74-ae98d3ceb2f3 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 01:06:14.049923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:14.049923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:14.049923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:14.049923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:14.049923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:14.049923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:14.049923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:14.049923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:14.049923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:14.049923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:14.049923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:14.049923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:14.049923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:14.049923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:14.049923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:14.051136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:14.051136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:14.051136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:14.051136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:14.051136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:14.051136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:14.051136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:14.051136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:14.051136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:14.051136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:14.051136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:14.051136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:14.051136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:14.051136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:14.051136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:14.051136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:14.051136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:14.051136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:14.052316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:14.052316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:14.052316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:14.052316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:14.052316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:14.052316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:14.052316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:14.052316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:14.052316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 01:06:14.052316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:14.054529 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 371/751] 10.222.0.15 () {56 vars in 1093 bytes} [Tue Sep 13 01:06:13 2022] GET /identity/v3/users/nova => generated 82 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:14.059206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6e8518-0774-45d3-8e74-ae98d3ceb2f3 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:14.114011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90beaba8-7fb6-4e98-87ea-f1306887a238 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:14.132650 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:14.152724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad713870-de40-4d0c-8af0-a22d41debeca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:14.153102 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad713870-de40-4d0c-8af0-a22d41debeca None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:14.153470 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad713870-de40-4d0c-8af0-a22d41debeca None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:14.153835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad713870-de40-4d0c-8af0-a22d41debeca None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:14.154723 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad713870-de40-4d0c-8af0-a22d41debeca None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:14.155070 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad713870-de40-4d0c-8af0-a22d41debeca None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:14.156609 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad713870-de40-4d0c-8af0-a22d41debeca None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:14.174689 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 381/752] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:14.174689 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad713870-de40-4d0c-8af0-a22d41debeca None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:14.188908 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:14.199466 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6e8518-0774-45d3-8e74-ae98d3ceb2f3 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:14.236336 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e36d88-f241-4498-b5e6-13f774bc5cab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:14.237074 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e36d88-f241-4498-b5e6-13f774bc5cab None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:14.237442 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e36d88-f241-4498-b5e6-13f774bc5cab None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:14.237807 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e36d88-f241-4498-b5e6-13f774bc5cab None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:14.238743 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e36d88-f241-4498-b5e6-13f774bc5cab None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:14.239164 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e36d88-f241-4498-b5e6-13f774bc5cab None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:14.241412 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e36d88-f241-4498-b5e6-13f774bc5cab None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:14.260843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 372/753] 10.222.0.15 () {56 vars in 1149 bytes} [Tue Sep 13 01:06:14 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:14.261888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e36d88-f241-4498-b5e6-13f774bc5cab None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:14.310316 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad713870-de40-4d0c-8af0-a22d41debeca None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:14.358426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89cb67c-b3e0-45f7-afd6-608638676f2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:14.359560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89cb67c-b3e0-45f7-afd6-608638676f2c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:14.360870 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89cb67c-b3e0-45f7-afd6-608638676f2c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:14.361393 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89cb67c-b3e0-45f7-afd6-608638676f2c None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:14.363326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89cb67c-b3e0-45f7-afd6-608638676f2c None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:14.363734 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89cb67c-b3e0-45f7-afd6-608638676f2c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:14.366215 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89cb67c-b3e0-45f7-afd6-608638676f2c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:14.375306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e36d88-f241-4498-b5e6-13f774bc5cab None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:14.396296 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 382/754] 10.222.0.15 () {56 vars in 1133 bytes} [Tue Sep 13 01:06:14 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:14.396296 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89cb67c-b3e0-45f7-afd6-608638676f2c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:14.422756 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5feb76-dc01-4609-8b0e-1b4337edd99f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:14.423596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5feb76-dc01-4609-8b0e-1b4337edd99f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:14.424025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5feb76-dc01-4609-8b0e-1b4337edd99f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:14.424892 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5feb76-dc01-4609-8b0e-1b4337edd99f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:14.426085 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5feb76-dc01-4609-8b0e-1b4337edd99f None admin] RBAC: Adding query filter params (role.id=66a36c858aab40688080eed6ceace3a0, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=e0d7d8ed4ede4ddca192fce0a497689d) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:14.426551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5feb76-dc01-4609-8b0e-1b4337edd99f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:14.428784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5feb76-dc01-4609-8b0e-1b4337edd99f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:14.436491 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 373/755] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:06:14 2022] GET /identity/v3/role_assignments?user.id=e0d7d8ed4ede4ddca192fce0a497689d&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=66a36c858aab40688080eed6ceace3a0 => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:14.442489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5feb76-dc01-4609-8b0e-1b4337edd99f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:14.444467 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:14.475970 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:14.564387 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89cb67c-b3e0-45f7-afd6-608638676f2c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:14.613267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178b4de4-8049-41e3-809b-93fe8c70db37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:14.614275 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178b4de4-8049-41e3-809b-93fe8c70db37 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:14.614796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178b4de4-8049-41e3-809b-93fe8c70db37 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:14.615318 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178b4de4-8049-41e3-809b-93fe8c70db37 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:14.655314 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5feb76-dc01-4609-8b0e-1b4337edd99f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:14.699656 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a38ac7-2a13-4fb4-a022-b80db0464420 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:14.701464 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a38ac7-2a13-4fb4-a022-b80db0464420 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:14.702057 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a38ac7-2a13-4fb4-a022-b80db0464420 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:14.702538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a38ac7-2a13-4fb4-a022-b80db0464420 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:14.717153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a38ac7-2a13-4fb4-a022-b80db0464420 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:14.719413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a38ac7-2a13-4fb4-a022-b80db0464420 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:14.740173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178b4de4-8049-41e3-809b-93fe8c70db37 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:14.741154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-88a38ac7-2a13-4fb4-a022-b80db0464420 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:14.741154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:14.741154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:14.741154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:14.741154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:14.741154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:14.741154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:14.741154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:14.741154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:14.741154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:14.741154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:14.741154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:14.741154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:14.741154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:14.741154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:14.741154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:14.742048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:14.742048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:14.742048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:14.742048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:14.742048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:14.742048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:14.742048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:14.742048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:14.742048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:14.742048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:14.742048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:14.742048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:14.742048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:14.742048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:14.742048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:14.742048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:14.742048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:14.742974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:14.742974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:14.742974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:14.743145 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178b4de4-8049-41e3-809b-93fe8c70db37 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:14.751525 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 374/756] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:14.751525 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a38ac7-2a13-4fb4-a022-b80db0464420 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:14.808142 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:14.867498 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:14.928076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 383/757] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:06:14 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 532 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 01:06:14.930925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a38ac7-2a13-4fb4-a022-b80db0464420 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:14.933697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178b4de4-8049-41e3-809b-93fe8c70db37 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:14.998626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aad103a-a3df-41eb-8b0a-3a49ec10cd90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:14.999637 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aad103a-a3df-41eb-8b0a-3a49ec10cd90 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:15.000146 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aad103a-a3df-41eb-8b0a-3a49ec10cd90 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:15.000628 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aad103a-a3df-41eb-8b0a-3a49ec10cd90 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:15.001488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aad103a-a3df-41eb-8b0a-3a49ec10cd90 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:15.002020 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aad103a-a3df-41eb-8b0a-3a49ec10cd90 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:15.003613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aad103a-a3df-41eb-8b0a-3a49ec10cd90 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:15.017011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 375/758] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:15.024726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aad103a-a3df-41eb-8b0a-3a49ec10cd90 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:15.066081 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178b4de4-8049-41e3-809b-93fe8c70db37 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:15.101194 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4b0abe-7f50-43c9-b54b-a38147f319dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:15.101825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4b0abe-7f50-43c9-b54b-a38147f319dd None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:15.102105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4b0abe-7f50-43c9-b54b-a38147f319dd None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:15.102375 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4b0abe-7f50-43c9-b54b-a38147f319dd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:15.110281 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:15.112934 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4b0abe-7f50-43c9-b54b-a38147f319dd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:15.114396 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4b0abe-7f50-43c9-b54b-a38147f319dd None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:15.130176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-db4b0abe-7f50-43c9-b54b-a38147f319dd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:15.130176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:15.130176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:15.130176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:15.130176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:15.130176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:15.130176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:15.130176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:15.130176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:15.130176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:15.130176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:15.130176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:15.130176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:15.130176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:15.130176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:15.130176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:15.131183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:15.131183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:15.131183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:15.131183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:15.131183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:15.131183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:15.131183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:15.131183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:15.131183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:15.131183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:15.131183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:15.131183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:15.131183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:15.131183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:15.131183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:15.131183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:15.131183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:15.132842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:15.132842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:15.132842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:15.132842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 384/759] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:15.138565 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4b0abe-7f50-43c9-b54b-a38147f319dd None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:15.252995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aad103a-a3df-41eb-8b0a-3a49ec10cd90 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:15.258879 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4b0abe-7f50-43c9-b54b-a38147f319dd None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:15.294805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988c9223-67bf-4fef-b620-cbbf3b7182ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:15.295605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988c9223-67bf-4fef-b620-cbbf3b7182ba None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:15.296404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988c9223-67bf-4fef-b620-cbbf3b7182ba None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:15.296778 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988c9223-67bf-4fef-b620-cbbf3b7182ba None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:15.297510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988c9223-67bf-4fef-b620-cbbf3b7182ba None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:15.297906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988c9223-67bf-4fef-b620-cbbf3b7182ba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:15.299427 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988c9223-67bf-4fef-b620-cbbf3b7182ba None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:15.314819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:15.315010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 385/760] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:15.320507 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988c9223-67bf-4fef-b620-cbbf3b7182ba None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:15.337197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8231f8d-3839-422b-9ca1-3cebef34c7d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:15.338194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8231f8d-3839-422b-9ca1-3cebef34c7d7 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:15.338703 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8231f8d-3839-422b-9ca1-3cebef34c7d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:15.339213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8231f8d-3839-422b-9ca1-3cebef34c7d7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:15.354032 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8231f8d-3839-422b-9ca1-3cebef34c7d7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:15.357072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8231f8d-3839-422b-9ca1-3cebef34c7d7 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:15.383153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-c8231f8d-3839-422b-9ca1-3cebef34c7d7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:15.383153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:15.383153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:15.383153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:15.383153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:15.383153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:15.383153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:15.383153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:15.383153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:15.383153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:15.383153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:15.383153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:15.383153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:15.383153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:15.383153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:15.384069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:15.384069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:15.384069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:15.384069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:15.384069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:15.384069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:15.384069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:15.384069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:15.384069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:15.384069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:15.384069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:15.384069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:15.384069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:15.384069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:15.384069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:15.384069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:15.384069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:15.384952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:15.384952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:15.384952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:15.384952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:15.384952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:15.384952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:15.384952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:15.384952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:15.387574 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 376/761] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 363 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:15.393833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8231f8d-3839-422b-9ca1-3cebef34c7d7 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:15.427219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:15.525122 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988c9223-67bf-4fef-b620-cbbf3b7182ba None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:15.597351 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa486400-18d6-4544-a372-d45d3e5ef0f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:15.599003 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa486400-18d6-4544-a372-d45d3e5ef0f3 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:15.599661 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa486400-18d6-4544-a372-d45d3e5ef0f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:15.601826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8231f8d-3839-422b-9ca1-3cebef34c7d7 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:15.602321 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa486400-18d6-4544-a372-d45d3e5ef0f3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:15.617817 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa486400-18d6-4544-a372-d45d3e5ef0f3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:15.620153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa486400-18d6-4544-a372-d45d3e5ef0f3 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:15.639853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-fa486400-18d6-4544-a372-d45d3e5ef0f3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:15.639853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:15.639853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:15.639853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:15.639853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:15.639853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:15.639853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:15.639853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:15.639853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:15.639853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:15.639853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:15.639853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:15.639853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:15.639853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:15.639853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:15.640769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:15.640769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:15.640769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:15.640769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:15.640769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:15.640769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:15.640769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:15.640769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:15.640769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:15.640769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:15.640769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:15.640769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:15.640769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:15.640769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:15.640769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:15.640769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:15.640769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:15.641685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:15.641685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:15.641685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:15.641685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:15.641685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:15.641685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:15.641685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:15.641685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:15.644048 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 386/762] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 324 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:15.652289 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa486400-18d6-4544-a372-d45d3e5ef0f3 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:15.686889 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11355528-6687-4fee-bee1-c9da26011fe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:15.687521 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11355528-6687-4fee-bee1-c9da26011fe0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:15.687799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11355528-6687-4fee-bee1-c9da26011fe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:15.688332 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11355528-6687-4fee-bee1-c9da26011fe0 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:15.689040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11355528-6687-4fee-bee1-c9da26011fe0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:15.689344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11355528-6687-4fee-bee1-c9da26011fe0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:15.690821 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11355528-6687-4fee-bee1-c9da26011fe0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:15.709925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 377/763] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:15.715782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11355528-6687-4fee-bee1-c9da26011fe0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:15.817952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:15.856494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa486400-18d6-4544-a372-d45d3e5ef0f3 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:15.871238 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:15.900298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38eb0ca6-4a5c-428f-b678-fd1506935429 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:15.900919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38eb0ca6-4a5c-428f-b678-fd1506935429 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:15.901197 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38eb0ca6-4a5c-428f-b678-fd1506935429 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:15.901468 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38eb0ca6-4a5c-428f-b678-fd1506935429 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:15.902158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38eb0ca6-4a5c-428f-b678-fd1506935429 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:15.902460 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38eb0ca6-4a5c-428f-b678-fd1506935429 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:15.903932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38eb0ca6-4a5c-428f-b678-fd1506935429 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:15.923099 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 387/764] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:15.924482 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38eb0ca6-4a5c-428f-b678-fd1506935429 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:15.945486 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11355528-6687-4fee-bee1-c9da26011fe0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:16.001241 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c22f516-69e2-4456-aac2-b4344393fee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:16.002464 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c22f516-69e2-4456-aac2-b4344393fee9 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:16.003005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c22f516-69e2-4456-aac2-b4344393fee9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:16.003533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c22f516-69e2-4456-aac2-b4344393fee9 None admin] PATH_INFO: `/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/2f2da4d2d3604d16b58dc25b8c17c49a/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:16.005646 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c22f516-69e2-4456-aac2-b4344393fee9 None admin] RBAC: Authorizing `identity:create_grant(project_id=e5c1519fdae742f4a25eb7cd5476a8cb, user_id=2f2da4d2d3604d16b58dc25b8c17c49a, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:16.008146 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c22f516-69e2-4456-aac2-b4344393fee9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:16.024865 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 378/765] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:06:15 2022] PUT /identity/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/2f2da4d2d3604d16b58dc25b8c17c49a/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:06:16.026477 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c185e906-3dc0-4df9-8abf-d9e4ac58e92a None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:16.027032 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c185e906-3dc0-4df9-8abf-d9e4ac58e92a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:16.027577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c185e906-3dc0-4df9-8abf-d9e4ac58e92a None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:16.033732 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 379/766] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:16.035831 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2535845-929b-46e1-8aab-4613f263932b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:16.036429 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2535845-929b-46e1-8aab-4613f263932b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:16.036910 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2535845-929b-46e1-8aab-4613f263932b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:16.060062 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c2535845-929b-46e1-8aab-4613f263932b None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:16.061803 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c2535845-929b-46e1-8aab-4613f263932b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:16.074986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38eb0ca6-4a5c-428f-b678-fd1506935429 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:16.121709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e5d461-7e77-46f6-b10b-73ab211b15af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:16.122331 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e5d461-7e77-46f6-b10b-73ab211b15af None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:16.122609 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e5d461-7e77-46f6-b10b-73ab211b15af None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:16.122930 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e5d461-7e77-46f6-b10b-73ab211b15af None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:16.123658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e5d461-7e77-46f6-b10b-73ab211b15af None admin] RBAC: Adding query filter params (role.id=71b1e7c4b5af44ddbb7a517760ad0119, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=e28a0ba45389417ba8ae92376dba87ff) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:16.123968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e5d461-7e77-46f6-b10b-73ab211b15af None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:16.125425 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e5d461-7e77-46f6-b10b-73ab211b15af None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:16.139181 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 388/767] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:06:15 2022] GET /identity/v3/role_assignments?user.id=e28a0ba45389417ba8ae92376dba87ff&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=71b1e7c4b5af44ddbb7a517760ad0119 => generated 259 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:16.141053 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2dc978-d45c-4452-94f6-0f3ee90d4f92 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:16.141053 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2dc978-d45c-4452-94f6-0f3ee90d4f92 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:16.141345 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2dc978-d45c-4452-94f6-0f3ee90d4f92 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:16.146962 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 389/768] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:16.152698 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773e0108-d9bf-4062-9fa1-27af640b70db None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:16.153014 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773e0108-d9bf-4062-9fa1-27af640b70db None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:16.153257 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773e0108-d9bf-4062-9fa1-27af640b70db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:16.198555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-773e0108-d9bf-4062-9fa1-27af640b70db None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:16.198555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-773e0108-d9bf-4062-9fa1-27af640b70db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:16.199185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 380/769] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:16 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:16.236265 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:16.285892 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 390/770] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:16 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 01:06:16.519255 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2535845-929b-46e1-8aab-4613f263932b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:16.556028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:16.590250 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773e0108-d9bf-4062-9fa1-27af640b70db None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:16.650526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2535845-929b-46e1-8aab-4613f263932b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:16.719235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed72e4f1-68d6-4d25-a8fb-dbab8ef435d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:16.719782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed72e4f1-68d6-4d25-a8fb-dbab8ef435d0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:16.720061 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed72e4f1-68d6-4d25-a8fb-dbab8ef435d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:16.720330 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed72e4f1-68d6-4d25-a8fb-dbab8ef435d0 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:16.720910 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed72e4f1-68d6-4d25-a8fb-dbab8ef435d0 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:16.722268 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed72e4f1-68d6-4d25-a8fb-dbab8ef435d0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:16.741216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-ed72e4f1-68d6-4d25-a8fb-dbab8ef435d0 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 01:06:16.741216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:16.741216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:16.741216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:16.741216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:16.741216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:16.741216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:16.741216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:16.741216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 01:06:16.741216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:16.741216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:16.741216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:16.741216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:16.741216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:16.741216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:16.741216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:16.741216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:16.742258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:16.742258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:16.742258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:16.742258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:16.742258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:16.742258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:16.742258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:16.742258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:16.742258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:16.742258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:16.742258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:16.742258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:16.742258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:16.742258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:16.742258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:16.742258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:16.743693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:16.743693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:16.743693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:16.743693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:16.743693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:16.743693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:16.743693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:16.743693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:16.743693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:16.743693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:16.743693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:16.743693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:16.743693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 01:06:16.743693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:16.743693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 381/771] 10.222.0.15 () {56 vars in 1129 bytes} [Tue Sep 13 01:06:16 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:16.750934 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed72e4f1-68d6-4d25-a8fb-dbab8ef435d0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:16.779974 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773e0108-d9bf-4062-9fa1-27af640b70db None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:16.855596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddd2af8-957e-4b24-b19d-e633789e66ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:16.856249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddd2af8-957e-4b24-b19d-e633789e66ac None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:16.856531 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddd2af8-957e-4b24-b19d-e633789e66ac None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:16.857073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddd2af8-957e-4b24-b19d-e633789e66ac None admin] PATH_INFO: `/v3/services/network` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:16.857383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eddd2af8-957e-4b24-b19d-e633789e66ac None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:16.858752 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eddd2af8-957e-4b24-b19d-e633789e66ac None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:16.877299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-eddd2af8-957e-4b24-b19d-e633789e66ac None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 01:06:16.877299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:16.877299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:16.877299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:16.877299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:16.877299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:16.877299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:16.877299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:16.877299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 01:06:16.877299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:16.877299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:16.877299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:16.877299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:16.877299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:16.877299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:16.877299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:16.877299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:16.878355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:16.878355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:16.878355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:16.878355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:16.878355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:16.878355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:16.878355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:16.878355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:16.878355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:16.878355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:16.878355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:16.878355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:16.878355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:16.878355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:16.878355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:16.878355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:16.879854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:16.879854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:16.879854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:16.879854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:16.879854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:16.879854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:16.879854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:16.879854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:16.879854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:16.879854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:16.879854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:16.879854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:16.879854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 01:06:16.879854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:16.879854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 391/772] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:16 2022] GET /identity/v3/services/network => generated 88 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:16.886168 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddd2af8-957e-4b24-b19d-e633789e66ac None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:16.888845 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed72e4f1-68d6-4d25-a8fb-dbab8ef435d0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:16.903292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:16.942087 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0acd10-fa29-4f1d-b008-e86471fe21ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:16.942708 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0acd10-fa29-4f1d-b008-e86471fe21ea None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:16.943025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0acd10-fa29-4f1d-b008-e86471fe21ea None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:16.943222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:16.943327 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0acd10-fa29-4f1d-b008-e86471fe21ea None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:16.944023 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0acd10-fa29-4f1d-b008-e86471fe21ea None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:16.944363 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0acd10-fa29-4f1d-b008-e86471fe21ea None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:16.946609 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0acd10-fa29-4f1d-b008-e86471fe21ea None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:16.957776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 382/773] 10.222.0.15 () {56 vars in 1124 bytes} [Tue Sep 13 01:06:16 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:16.963746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0acd10-fa29-4f1d-b008-e86471fe21ea None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:17.092777 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddd2af8-957e-4b24-b19d-e633789e66ac None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:17.138092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0acd10-fa29-4f1d-b008-e86471fe21ea None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:17.169716 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185df7ad-81a4-47ed-ba66-16adebb0dc2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:17.170341 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185df7ad-81a4-47ed-ba66-16adebb0dc2f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:17.170627 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185df7ad-81a4-47ed-ba66-16adebb0dc2f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:17.170919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185df7ad-81a4-47ed-ba66-16adebb0dc2f None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:17.171558 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185df7ad-81a4-47ed-ba66-16adebb0dc2f None admin] RBAC: Adding query filter params (name=network) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:17.171860 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185df7ad-81a4-47ed-ba66-16adebb0dc2f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:17.174163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185df7ad-81a4-47ed-ba66-16adebb0dc2f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:17.189225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 392/774] 10.222.0.15 () {56 vars in 1112 bytes} [Tue Sep 13 01:06:16 2022] GET /identity/v3/services?name=network => generated 124 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:17.190912 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cab2ba-f9a7-496f-9140-2df9465eb657 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:17.191201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cab2ba-f9a7-496f-9140-2df9465eb657 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:17.191476 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cab2ba-f9a7-496f-9140-2df9465eb657 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:17.194552 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 393/775] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:17.196305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21448f46-6108-408d-a56e-a2023ace8d60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:17.196924 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21448f46-6108-408d-a56e-a2023ace8d60 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:17.197207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21448f46-6108-408d-a56e-a2023ace8d60 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:17.197476 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21448f46-6108-408d-a56e-a2023ace8d60 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:17.198106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21448f46-6108-408d-a56e-a2023ace8d60 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:17.198409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21448f46-6108-408d-a56e-a2023ace8d60 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:17.198693 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cab2ba-f9a7-496f-9140-2df9465eb657 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:17.199848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21448f46-6108-408d-a56e-a2023ace8d60 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:17.206569 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 383/776] 10.222.0.15 () {56 vars in 1124 bytes} [Tue Sep 13 01:06:16 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:17.207801 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2858be53-b274-447b-b85a-8805b4c43f25 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:17.208196 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2858be53-b274-447b-b85a-8805b4c43f25 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:17.208477 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2858be53-b274-447b-b85a-8805b4c43f25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:17.229043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-2858be53-b274-447b-b85a-8805b4c43f25 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:17.230352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2858be53-b274-447b-b85a-8805b4c43f25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:17.259234 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:17.309116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:17.323134 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 384/777] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:17 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:17.324407 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cab2ba-f9a7-496f-9140-2df9465eb657 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:17.326049 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2858be53-b274-447b-b85a-8805b4c43f25 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:17.396595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901271de-e48a-4fa1-94d7-8fb3adff9291 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:17.397192 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901271de-e48a-4fa1-94d7-8fb3adff9291 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:17.397471 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901271de-e48a-4fa1-94d7-8fb3adff9291 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:17.397748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901271de-e48a-4fa1-94d7-8fb3adff9291 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:17.398376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901271de-e48a-4fa1-94d7-8fb3adff9291 None admin] RBAC: Adding query filter params (type=network) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:17.398679 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901271de-e48a-4fa1-94d7-8fb3adff9291 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:17.400056 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901271de-e48a-4fa1-94d7-8fb3adff9291 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:17.421091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 394/778] 10.222.0.15 () {56 vars in 1112 bytes} [Tue Sep 13 01:06:17 2022] GET /identity/v3/services?type=network => generated 350 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:17.428485 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901271de-e48a-4fa1-94d7-8fb3adff9291 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:17.512055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2858be53-b274-447b-b85a-8805b4c43f25 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:17.553955 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314f20b0-47f9-4718-a9d3-40cc5896c104 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:17.555292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314f20b0-47f9-4718-a9d3-40cc5896c104 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:17.555825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314f20b0-47f9-4718-a9d3-40cc5896c104 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:17.556252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314f20b0-47f9-4718-a9d3-40cc5896c104 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:17.557097 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314f20b0-47f9-4718-a9d3-40cc5896c104 None admin] RBAC: Adding query filter params (interface=public, service_id=252fd5ebe5ff4f08898511e2824db07a, region_id=RegionOne) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:17.557564 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314f20b0-47f9-4718-a9d3-40cc5896c104 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:17.559167 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314f20b0-47f9-4718-a9d3-40cc5896c104 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:17.569487 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 385/779] 10.222.0.15 () {56 vars in 1251 bytes} [Tue Sep 13 01:06:17 2022] GET /identity/v3/endpoints?service_id=252fd5ebe5ff4f08898511e2824db07a&interface=public®ion_id=RegionOne => generated 194 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:17.575769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314f20b0-47f9-4718-a9d3-40cc5896c104 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:17.612136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:17.612462 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901271de-e48a-4fa1-94d7-8fb3adff9291 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:17.649492 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ea93ea-6e62-4a3a-bbbb-f98c4bd72387 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:17.650017 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ea93ea-6e62-4a3a-bbbb-f98c4bd72387 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:17.650293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ea93ea-6e62-4a3a-bbbb-f98c4bd72387 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:17.650564 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ea93ea-6e62-4a3a-bbbb-f98c4bd72387 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:17.651286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ea93ea-6e62-4a3a-bbbb-f98c4bd72387 None admin] RBAC: Adding query filter params (interface=public, service_id=3604ac50509e4222919c83c621c30fd2, region_id=RegionOne) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:17.651594 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ea93ea-6e62-4a3a-bbbb-f98c4bd72387 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:17.652936 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ea93ea-6e62-4a3a-bbbb-f98c4bd72387 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:17.666886 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 395/780] 10.222.0.15 () {56 vars in 1251 bytes} [Tue Sep 13 01:06:17 2022] GET /identity/v3/endpoints?service_id=3604ac50509e4222919c83c621c30fd2&interface=public®ion_id=RegionOne => generated 194 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:17.666886 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ea93ea-6e62-4a3a-bbbb-f98c4bd72387 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:17.684644 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:17.716576 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314f20b0-47f9-4718-a9d3-40cc5896c104 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:17.760654 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4724fc3a-9392-42d9-8248-a95373ca2a86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:17.762120 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4724fc3a-9392-42d9-8248-a95373ca2a86 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:17.762703 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4724fc3a-9392-42d9-8248-a95373ca2a86 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:17.763197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4724fc3a-9392-42d9-8248-a95373ca2a86 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:17.764592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4724fc3a-9392-42d9-8248-a95373ca2a86 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:17.766671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4724fc3a-9392-42d9-8248-a95373ca2a86 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:17.770194 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ea93ea-6e62-4a3a-bbbb-f98c4bd72387 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:17.774838 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 386/781] 10.222.0.15 () {56 vars in 1087 bytes} [Tue Sep 13 01:06:17 2022] GET /identity/v3/services => generated 772 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:17.776046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4724fc3a-9392-42d9-8248-a95373ca2a86 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:17.820016 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a819dcc7-7404-4606-b420-bf0b27a1c18a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:17.820721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a819dcc7-7404-4606-b420-bf0b27a1c18a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:17.821001 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a819dcc7-7404-4606-b420-bf0b27a1c18a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:17.821273 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a819dcc7-7404-4606-b420-bf0b27a1c18a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:17.842202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a819dcc7-7404-4606-b420-bf0b27a1c18a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:17.854884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-a819dcc7-7404-4606-b420-bf0b27a1c18a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:06:17.854884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:17.854884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:17.854884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:17.854884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:17.854884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:17.854884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:17.854884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:17.854884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:17.854884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:17.854884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:17.854884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:17.854884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:17.854884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:17.854884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:17.854884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:17.855881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:17.855881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:17.855881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:17.855881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:17.855881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:17.855881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:17.855881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:17.855881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:17.855881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:17.855881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:17.855881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:17.855881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:17.855881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:17.855881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:17.855881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:17.855881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:17.857131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:17.857131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:17.857131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:17.857131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:17.857131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:17.857131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:17.857131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:17.857131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:17.857131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:17.857131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:17.857131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:06:17.857131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:17.858246 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 396/782] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:06:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:17.864187 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a819dcc7-7404-4606-b420-bf0b27a1c18a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:17.943280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:17.972407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4724fc3a-9392-42d9-8248-a95373ca2a86 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:18.016281 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:18.023999 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb7c3f1-76be-43b7-bb0d-09d3f25ca87a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:18.025371 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb7c3f1-76be-43b7-bb0d-09d3f25ca87a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:18.026117 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb7c3f1-76be-43b7-bb0d-09d3f25ca87a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:18.026592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb7c3f1-76be-43b7-bb0d-09d3f25ca87a None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:18.027606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb7c3f1-76be-43b7-bb0d-09d3f25ca87a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:18.029818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb7c3f1-76be-43b7-bb0d-09d3f25ca87a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:18.037315 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 387/783] 10.222.0.15 () {56 vars in 1087 bytes} [Tue Sep 13 01:06:17 2022] GET /identity/v3/services => generated 772 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:18.040729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a819dcc7-7404-4606-b420-bf0b27a1c18a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:18.079778 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f234ca-550e-46b5-bac0-681033195fd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:18.080398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f234ca-550e-46b5-bac0-681033195fd8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:18.080675 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f234ca-550e-46b5-bac0-681033195fd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:18.080946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f234ca-550e-46b5-bac0-681033195fd8 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:18.081615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f234ca-550e-46b5-bac0-681033195fd8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:18.081918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f234ca-550e-46b5-bac0-681033195fd8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:18.083351 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f234ca-550e-46b5-bac0-681033195fd8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:18.093409 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 397/784] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:06:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:18.099224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb7c3f1-76be-43b7-bb0d-09d3f25ca87a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:18.223552 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb7c3f1-76be-43b7-bb0d-09d3f25ca87a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:18.272853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7968a8-7038-452b-8243-7801671f2ac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:18.273500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7968a8-7038-452b-8243-7801671f2ac4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:18.273780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7968a8-7038-452b-8243-7801671f2ac4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:18.274052 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7968a8-7038-452b-8243-7801671f2ac4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:18.290677 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7968a8-7038-452b-8243-7801671f2ac4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:18.292252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7968a8-7038-452b-8243-7801671f2ac4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:18.307544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-0f7968a8-7038-452b-8243-7801671f2ac4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:18.307544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:18.307544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:18.307544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:18.307544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:18.307544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:18.307544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:18.307544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:18.307544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:18.307544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:18.307544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:18.307544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:18.307544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:18.307544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:18.307544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:18.307544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:18.308504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:18.308504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:18.308504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:18.308504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:18.308504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:18.308504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:18.308504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:18.308504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:18.308504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:18.308504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:18.308504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:18.308504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:18.308504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:18.308504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:18.308504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:18.308504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:18.308504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:18.309405 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:18.309405 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:18.309405 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:18.315398 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 388/785] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:18.321621 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f234ca-550e-46b5-bac0-681033195fd8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:18.447120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f234ca-550e-46b5-bac0-681033195fd8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:18.489085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3ee7c9-697a-4dc1-b844-2a60321ea12f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:18.489631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3ee7c9-697a-4dc1-b844-2a60321ea12f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:18.489908 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3ee7c9-697a-4dc1-b844-2a60321ea12f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:18.490176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3ee7c9-697a-4dc1-b844-2a60321ea12f None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:18.491367 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3ee7c9-697a-4dc1-b844-2a60321ea12f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:18.491670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3ee7c9-697a-4dc1-b844-2a60321ea12f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:18.493029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3ee7c9-697a-4dc1-b844-2a60321ea12f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:18.502622 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:18.510967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 398/786] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:18.512634 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7968a8-7038-452b-8243-7801671f2ac4 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:18.517198 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:18.570390 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3b6ade-fdc5-4275-9cfd-1229167ab4b6 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:18.571100 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3b6ade-fdc5-4275-9cfd-1229167ab4b6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:18.571384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3b6ade-fdc5-4275-9cfd-1229167ab4b6 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:18.573612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 399/787] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:18 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:18.580391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af06e039-6b57-4124-a17b-4e903c5c5bc2 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:18.580968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af06e039-6b57-4124-a17b-4e903c5c5bc2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:18.581249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af06e039-6b57-4124-a17b-4e903c5c5bc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:18.621076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-af06e039-6b57-4124-a17b-4e903c5c5bc2 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:18.622643 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-af06e039-6b57-4124-a17b-4e903c5c5bc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:18.677075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7968a8-7038-452b-8243-7801671f2ac4 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:18.724148 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 400/788] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:18 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 01:06:18.748655 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e167c4-43f2-4461-b0f4-aafff86e2b25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:18.749234 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e167c4-43f2-4461-b0f4-aafff86e2b25 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:18.749518 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e167c4-43f2-4461-b0f4-aafff86e2b25 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:18.749795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e167c4-43f2-4461-b0f4-aafff86e2b25 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:18.767998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e167c4-43f2-4461-b0f4-aafff86e2b25 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:18.769447 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e167c4-43f2-4461-b0f4-aafff86e2b25 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:18.781431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-13e167c4-43f2-4461-b0f4-aafff86e2b25 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 13 01:06:18.781431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:18.781431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:18.781431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:18.781431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:18.781431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:18.781431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:18.781431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:18.781431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:18.781431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:18.781431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:18.781431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:18.781431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:18.781431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:18.781431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:18.781431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:18.782377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:18.782377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:18.782377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:18.782377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:18.782377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:18.782377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:18.782377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:18.782377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:18.782377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:18.782377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:18.782377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:18.782377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:18.782377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:18.782377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:18.782377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:18.782377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:18.782377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:18.782377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:18.783340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:18.783340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:18.783340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:18.783340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:18.783340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:18.783340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:18.783340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:18.783340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:18.783340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 13 01:06:18.783340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:18.787015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 389/789] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:06:18 2022] GET /identity/v3/users/swift => generated 83 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:18.792969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af06e039-6b57-4124-a17b-4e903c5c5bc2 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:18.816136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7436c25d-3de6-418a-9f54-2a0fe49fb039 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:18.816457 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7436c25d-3de6-418a-9f54-2a0fe49fb039 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:18.816820 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7436c25d-3de6-418a-9f54-2a0fe49fb039 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:18.818274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 390/790] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:18.824263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2818335-5cd2-4e26-a52d-14653a320969 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:18.824778 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2818335-5cd2-4e26-a52d-14653a320969 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:18.825262 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2818335-5cd2-4e26-a52d-14653a320969 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:18.847052 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b2818335-5cd2-4e26-a52d-14653a320969 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:18.849520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b2818335-5cd2-4e26-a52d-14653a320969 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:18.923210 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:18.931855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 391/791] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:18 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 01:06:18.949281 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af06e039-6b57-4124-a17b-4e903c5c5bc2 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:18.979500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:18.985252 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99396d86-968a-488b-b5bc-bd183ee8b15a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:18.985855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99396d86-968a-488b-b5bc-bd183ee8b15a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:18.986136 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99396d86-968a-488b-b5bc-bd183ee8b15a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:18.986401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99396d86-968a-488b-b5bc-bd183ee8b15a None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:18.987162 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99396d86-968a-488b-b5bc-bd183ee8b15a None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:18.987462 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99396d86-968a-488b-b5bc-bd183ee8b15a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:18.988876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99396d86-968a-488b-b5bc-bd183ee8b15a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:19.008756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 401/792] 10.222.0.15 () {56 vars in 1135 bytes} [Tue Sep 13 01:06:18 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:19.015088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2818335-5cd2-4e26-a52d-14653a320969 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:19.074832 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99396d86-968a-488b-b5bc-bd183ee8b15a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:19.145728 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2818335-5cd2-4e26-a52d-14653a320969 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:19.192968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18726096-4eca-4626-8fa2-af177a08bf38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:19.195056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18726096-4eca-4626-8fa2-af177a08bf38 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:19.195806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18726096-4eca-4626-8fa2-af177a08bf38 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:19.196508 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18726096-4eca-4626-8fa2-af177a08bf38 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:19.228202 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18726096-4eca-4626-8fa2-af177a08bf38 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:19.234407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18726096-4eca-4626-8fa2-af177a08bf38 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:19.283811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-18726096-4eca-4626-8fa2-af177a08bf38 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:19.283811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:19.283811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:19.283811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:19.283811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:19.283811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:19.283811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:19.283811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:19.283811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:19.283811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:19.283811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:19.283811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:19.283811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:19.283811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:19.283811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:19.283811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:19.285536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:19.285536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:19.285536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:19.285536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:19.285536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:19.285536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:19.285536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:19.285536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:19.285536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:19.285536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:19.285536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:19.285536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:19.285536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:19.285536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:19.285536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:19.285536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:19.285536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:19.287345 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:19.287345 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:19.287345 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:19.292076 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 392/793] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:19.300978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18726096-4eca-4626-8fa2-af177a08bf38 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:19.322263 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99396d86-968a-488b-b5bc-bd183ee8b15a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:19.374687 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:19.392010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c938b9a8-018a-4eb9-8afa-b5fa471e8611 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:19.394351 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c938b9a8-018a-4eb9-8afa-b5fa471e8611 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:19.395479 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c938b9a8-018a-4eb9-8afa-b5fa471e8611 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:19.397039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c938b9a8-018a-4eb9-8afa-b5fa471e8611 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:19.417457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c938b9a8-018a-4eb9-8afa-b5fa471e8611 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:19.423728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-c938b9a8-018a-4eb9-8afa-b5fa471e8611 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:06:19.423728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:19.423728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:19.423728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:19.423728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:19.423728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:19.423728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:19.423728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:19.423728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:19.423728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:19.423728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:19.423728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:19.423728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:19.423728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:19.423728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:19.423728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:19.424652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:19.424652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:19.424652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:19.424652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:19.424652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:19.424652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:19.424652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:19.424652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:19.424652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:19.424652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:19.424652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:19.424652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:19.424652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:19.424652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:19.424652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:19.424652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:19.425478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:19.425478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:19.425478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:19.425478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:19.425478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:19.425478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:19.425478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:19.425478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:19.425478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:19.425478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:19.425478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:06:19.425478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:19.428731 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 402/794] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:06:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 354 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:19.429332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c938b9a8-018a-4eb9-8afa-b5fa471e8611 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:19.460794 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18726096-4eca-4626-8fa2-af177a08bf38 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:19.505985 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:19.507762 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b91da55-8a25-44c9-b776-257f1b466a1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:19.508755 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b91da55-8a25-44c9-b776-257f1b466a1b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:19.509292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b91da55-8a25-44c9-b776-257f1b466a1b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:19.510441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b91da55-8a25-44c9-b776-257f1b466a1b None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:19.511790 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b91da55-8a25-44c9-b776-257f1b466a1b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:19.512326 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b91da55-8a25-44c9-b776-257f1b466a1b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:19.515546 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b91da55-8a25-44c9-b776-257f1b466a1b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:19.533318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 393/795] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:19.535027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b91da55-8a25-44c9-b776-257f1b466a1b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:19.561181 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c938b9a8-018a-4eb9-8afa-b5fa471e8611 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:19.604049 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82fdf4e5-bb99-4090-84c3-8eaa09f1225f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:19.605389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fdf4e5-bb99-4090-84c3-8eaa09f1225f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:19.605917 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fdf4e5-bb99-4090-84c3-8eaa09f1225f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:19.606404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fdf4e5-bb99-4090-84c3-8eaa09f1225f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:19.622092 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82fdf4e5-bb99-4090-84c3-8eaa09f1225f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:19.628799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-82fdf4e5-bb99-4090-84c3-8eaa09f1225f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:06:19.628799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:19.628799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:19.628799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:19.628799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:19.628799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:19.628799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:19.628799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:19.628799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:19.628799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:19.628799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:19.628799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:19.628799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:19.628799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:19.628799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:19.628799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:19.629756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:19.629756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:19.629756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:19.629756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:19.629756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:19.629756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:19.629756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:19.629756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:19.629756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:19.629756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:19.629756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:19.629756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:19.629756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:19.629756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:19.629756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:19.629756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:19.630599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:19.630599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:19.630599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:19.630599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:19.630599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:19.630599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:19.630599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:19.630599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:19.630599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:19.630599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:19.630599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:06:19.630599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:19.634932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 403/796] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:06:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:19.635945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82fdf4e5-bb99-4090-84c3-8eaa09f1225f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:19.663941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b91da55-8a25-44c9-b776-257f1b466a1b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:19.667115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:19.701426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e806ba-b42b-4cfd-993b-0dc3e4b83ac2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:19.702034 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e806ba-b42b-4cfd-993b-0dc3e4b83ac2 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:19.702308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e806ba-b42b-4cfd-993b-0dc3e4b83ac2 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:19.702626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e806ba-b42b-4cfd-993b-0dc3e4b83ac2 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:19.703328 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e806ba-b42b-4cfd-993b-0dc3e4b83ac2 None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:19.703624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e806ba-b42b-4cfd-993b-0dc3e4b83ac2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:19.704983 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e806ba-b42b-4cfd-993b-0dc3e4b83ac2 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:19.713596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 394/797] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:06:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:19.714404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e806ba-b42b-4cfd-993b-0dc3e4b83ac2 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:19.775902 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82fdf4e5-bb99-4090-84c3-8eaa09f1225f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:19.789811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:19.822152 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e806ba-b42b-4cfd-993b-0dc3e4b83ac2 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:19.825696 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996b29da-4ac6-4733-a498-e2b512ca6329 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:19.826409 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996b29da-4ac6-4733-a498-e2b512ca6329 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:19.826765 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996b29da-4ac6-4733-a498-e2b512ca6329 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:19.827137 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996b29da-4ac6-4733-a498-e2b512ca6329 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:19.836154 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996b29da-4ac6-4733-a498-e2b512ca6329 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:19.838426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996b29da-4ac6-4733-a498-e2b512ca6329 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:19.852510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-996b29da-4ac6-4733-a498-e2b512ca6329 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:19.852510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:19.852510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:19.852510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:19.852510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:19.852510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:19.852510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:19.852510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:19.852510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:19.852510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:19.852510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:19.852510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:19.852510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:19.852510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:19.852510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:19.853846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:19.853846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:19.853846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:19.853846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:19.853846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:19.853846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:19.853846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:19.853846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:19.853846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:19.853846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:19.853846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:19.853846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:19.853846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:19.853846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:19.853846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:19.853846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:19.853846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:19.856255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:19.856255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:19.856255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:19.856255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:19.856255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:19.856255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:19.856255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:19.856255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:19.856255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 404/798] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:19.856255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996b29da-4ac6-4733-a498-e2b512ca6329 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:19.866880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270bcb3e-3515-43d6-be51-87cd7083fe5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:19.867521 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270bcb3e-3515-43d6-be51-87cd7083fe5d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:19.868112 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270bcb3e-3515-43d6-be51-87cd7083fe5d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:19.868404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270bcb3e-3515-43d6-be51-87cd7083fe5d None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:19.869359 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270bcb3e-3515-43d6-be51-87cd7083fe5d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:19.869668 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270bcb3e-3515-43d6-be51-87cd7083fe5d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:19.871593 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270bcb3e-3515-43d6-be51-87cd7083fe5d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:19.883315 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 395/799] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:06:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:19.884148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270bcb3e-3515-43d6-be51-87cd7083fe5d None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:19.920231 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:19.968407 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996b29da-4ac6-4733-a498-e2b512ca6329 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:20.005677 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b46647-2138-4687-b87e-b82b4ad76b36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:20.006553 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b46647-2138-4687-b87e-b82b4ad76b36 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:20.006833 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b46647-2138-4687-b87e-b82b4ad76b36 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:20.007142 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b46647-2138-4687-b87e-b82b4ad76b36 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:20.013769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270bcb3e-3515-43d6-be51-87cd7083fe5d None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:20.018574 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b46647-2138-4687-b87e-b82b4ad76b36 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:20.020209 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b46647-2138-4687-b87e-b82b4ad76b36 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:20.035905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-74b46647-2138-4687-b87e-b82b4ad76b36 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:20.035905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:20.035905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:20.035905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:20.035905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:20.035905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:20.035905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:20.035905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:20.035905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:20.035905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:20.035905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:20.035905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:20.035905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:20.035905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:20.035905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:20.035905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:20.037104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:20.037104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:20.037104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:20.037104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:20.037104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:20.037104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:20.037104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:20.037104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:20.037104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:20.037104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:20.037104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:20.037104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:20.037104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:20.037104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:20.037104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:20.037104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:20.037104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:20.038418 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:20.038418 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:20.038418 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:20.038418 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 405/800] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:20.038418 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b46647-2138-4687-b87e-b82b4ad76b36 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:20.059176 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb17f07f-8ce4-4fc7-aa6c-6a9ffdc38c81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:20.060643 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb17f07f-8ce4-4fc7-aa6c-6a9ffdc38c81 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:20.061078 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb17f07f-8ce4-4fc7-aa6c-6a9ffdc38c81 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:20.061502 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb17f07f-8ce4-4fc7-aa6c-6a9ffdc38c81 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:20.062360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb17f07f-8ce4-4fc7-aa6c-6a9ffdc38c81 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:20.062816 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb17f07f-8ce4-4fc7-aa6c-6a9ffdc38c81 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:20.063861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:20.065646 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb17f07f-8ce4-4fc7-aa6c-6a9ffdc38c81 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:20.086888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 396/801] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:20.086888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb17f07f-8ce4-4fc7-aa6c-6a9ffdc38c81 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:20.154864 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b46647-2138-4687-b87e-b82b4ad76b36 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:20.191387 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f00cead-9770-436c-af6a-978f3032d876 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:20.192016 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f00cead-9770-436c-af6a-978f3032d876 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:20.192291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f00cead-9770-436c-af6a-978f3032d876 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:20.192560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f00cead-9770-436c-af6a-978f3032d876 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:20.204182 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f00cead-9770-436c-af6a-978f3032d876 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:20.205649 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f00cead-9770-436c-af6a-978f3032d876 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:20.221139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-6f00cead-9770-436c-af6a-978f3032d876 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:20.221139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:20.221139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:20.221139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:20.221139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:20.221139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:20.221139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:20.221139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:20.221139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:20.221139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:20.221139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:20.221139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:20.221139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:20.221139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:20.221139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:20.221139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:20.222114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:20.222114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:20.222114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:20.222114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:20.222114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:20.222114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:20.222114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:20.222114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:20.222114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:20.222114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:20.222114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:20.222114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:20.222114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:20.222114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:20.222114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:20.222114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:20.222114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:20.223052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:20.223052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:20.223052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:20.223937 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb17f07f-8ce4-4fc7-aa6c-6a9ffdc38c81 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:20.227240 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:20.227362 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 406/802] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:20.228687 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f00cead-9770-436c-af6a-978f3032d876 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:20.270170 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f794b2c-99ea-4a1a-add9-18223f72ad49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:20.272223 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f794b2c-99ea-4a1a-add9-18223f72ad49 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:20.272741 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f794b2c-99ea-4a1a-add9-18223f72ad49 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:20.273219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f794b2c-99ea-4a1a-add9-18223f72ad49 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:20.274511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f794b2c-99ea-4a1a-add9-18223f72ad49 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:20.275050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f794b2c-99ea-4a1a-add9-18223f72ad49 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:20.277344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f794b2c-99ea-4a1a-add9-18223f72ad49 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:20.294333 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 397/803] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:20.296119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f794b2c-99ea-4a1a-add9-18223f72ad49 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:20.306234 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:20.340781 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f00cead-9770-436c-af6a-978f3032d876 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:20.381571 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b126286f-2014-4b9a-a776-0fc5a55a15ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:20.382222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b126286f-2014-4b9a-a776-0fc5a55a15ef None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:20.382493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b126286f-2014-4b9a-a776-0fc5a55a15ef None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:20.382755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b126286f-2014-4b9a-a776-0fc5a55a15ef None admin] PATH_INFO: `/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/e0d7d8ed4ede4ddca192fce0a497689d/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:20.384975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b126286f-2014-4b9a-a776-0fc5a55a15ef None admin] RBAC: Authorizing `identity:create_grant(project_id=e5c1519fdae742f4a25eb7cd5476a8cb, user_id=e0d7d8ed4ede4ddca192fce0a497689d, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:20.386466 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b126286f-2014-4b9a-a776-0fc5a55a15ef None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:20.407420 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 407/804] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:06:20 2022] PUT /identity/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/e0d7d8ed4ede4ddca192fce0a497689d/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:06:20.408740 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b126286f-2014-4b9a-a776-0fc5a55a15ef None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:20.462989 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f794b2c-99ea-4a1a-add9-18223f72ad49 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:20.510156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b71ab6-eada-4985-9614-598629750730 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:20.510786 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b71ab6-eada-4985-9614-598629750730 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:20.511093 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b71ab6-eada-4985-9614-598629750730 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:20.511365 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b71ab6-eada-4985-9614-598629750730 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:20.512025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b71ab6-eada-4985-9614-598629750730 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:20.512383 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b71ab6-eada-4985-9614-598629750730 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:20.513761 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b71ab6-eada-4985-9614-598629750730 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:20.528962 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 398/805] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:20.530364 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ad4e1e-113d-48e5-8512-9bc854b085c3 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:20.530767 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ad4e1e-113d-48e5-8512-9bc854b085c3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:20.531580 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ad4e1e-113d-48e5-8512-9bc854b085c3 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:20.536943 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 399/806] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:20 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:20.538817 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c6fabc-b8b7-4319-9a68-4133cac62b08 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:20.539152 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c6fabc-b8b7-4319-9a68-4133cac62b08 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:20.539430 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c6fabc-b8b7-4319-9a68-4133cac62b08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:20.560197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-73c6fabc-b8b7-4319-9a68-4133cac62b08 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:20.561422 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-73c6fabc-b8b7-4319-9a68-4133cac62b08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:20.564142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:20.567929 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b126286f-2014-4b9a-a776-0fc5a55a15ef None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:20.618713 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:20.627253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19aa829e-8d85-4724-a046-00afd6f58bc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:20.629009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19aa829e-8d85-4724-a046-00afd6f58bc9 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:20.629788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19aa829e-8d85-4724-a046-00afd6f58bc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:20.630284 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19aa829e-8d85-4724-a046-00afd6f58bc9 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:20.642825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19aa829e-8d85-4724-a046-00afd6f58bc9 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:20.644382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19aa829e-8d85-4724-a046-00afd6f58bc9 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:20.648108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 400/807] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:20 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:20.648789 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c6fabc-b8b7-4319-9a68-4133cac62b08 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:20.657560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-19aa829e-8d85-4724-a046-00afd6f58bc9 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 01:06:20.657560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:20.657560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:20.657560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:20.657560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:20.657560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:20.657560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:20.657560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:20.657560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:20.657560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:20.657560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:20.657560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:20.657560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:20.657560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:20.657560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:20.657560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:20.658504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:20.658504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:20.658504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:20.658504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:20.658504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:20.658504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:20.658504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:20.658504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:20.658504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:20.658504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:20.658504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:20.658504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:20.658504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:20.658504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:20.658504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:20.658504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:20.658504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:20.658504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:20.659484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:20.659484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:20.659484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:20.659484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:20.659484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:20.659484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:20.659484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:20.659484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:20.659484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 01:06:20.659484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:20.659484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 408/808] 10.222.0.15 () {56 vars in 1117 bytes} [Tue Sep 13 01:06:20 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:20.667279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19aa829e-8d85-4724-a046-00afd6f58bc9 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:20.768618 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c6fabc-b8b7-4319-9a68-4133cac62b08 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:20.800459 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19aa829e-8d85-4724-a046-00afd6f58bc9 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:20.809386 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49bf2951-a1e6-4524-8a64-a8ec084ba746 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:20.810033 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bf2951-a1e6-4524-8a64-a8ec084ba746 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:20.810310 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bf2951-a1e6-4524-8a64-a8ec084ba746 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:20.810580 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bf2951-a1e6-4524-8a64-a8ec084ba746 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:20.822259 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49bf2951-a1e6-4524-8a64-a8ec084ba746 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:20.823761 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49bf2951-a1e6-4524-8a64-a8ec084ba746 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:20.825747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:20.838886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-49bf2951-a1e6-4524-8a64-a8ec084ba746 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 01:06:20.838886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:20.838886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:20.838886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:20.838886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:20.838886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:20.838886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:20.838886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:20.838886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:20.838886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:20.838886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:20.838886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:20.838886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:20.838886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:20.838886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:20.838886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:20.839940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:20.839940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:20.839940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:20.839940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:20.839940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:20.839940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:20.839940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:20.839940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:20.839940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:20.839940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:20.839940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:20.839940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:20.839940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:20.839940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:20.839940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:20.839940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:20.839940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:20.839940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:20.840877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:20.840877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:20.840877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:20.840877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:20.840877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:20.840877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:20.840877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:20.840877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:20.840877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 01:06:20.840877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:20.840877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 401/809] 10.222.0.15 () {56 vars in 1093 bytes} [Tue Sep 13 01:06:20 2022] GET /identity/v3/users/nova => generated 82 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:20.841452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853b849c-bef8-4545-9eaf-483938d19eb6 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:20.843001 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853b849c-bef8-4545-9eaf-483938d19eb6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:20.843001 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853b849c-bef8-4545-9eaf-483938d19eb6 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:20.845502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1b4dcc-9c74-446a-b583-42a016893506 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:20.846110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1b4dcc-9c74-446a-b583-42a016893506 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:20.846383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1b4dcc-9c74-446a-b583-42a016893506 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:20.846650 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1b4dcc-9c74-446a-b583-42a016893506 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:20.847356 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 402/810] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:20.847481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1b4dcc-9c74-446a-b583-42a016893506 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:20.847718 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1b4dcc-9c74-446a-b583-42a016893506 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:20.848140 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853b849c-bef8-4545-9eaf-483938d19eb6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:20.849402 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1b4dcc-9c74-446a-b583-42a016893506 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:20.870443 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 409/811] 10.222.0.15 () {56 vars in 1149 bytes} [Tue Sep 13 01:06:20 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:20.871629 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2419e3c7-cd1d-4494-bf20-d32628acb113 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:20.871908 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2419e3c7-cd1d-4494-bf20-d32628acb113 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:20.872174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2419e3c7-cd1d-4494-bf20-d32628acb113 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:20.965181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853b849c-bef8-4545-9eaf-483938d19eb6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:21.003253 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970001d5-26db-40e7-94b2-86e995392f21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:21.003870 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970001d5-26db-40e7-94b2-86e995392f21 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:21.004153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970001d5-26db-40e7-94b2-86e995392f21 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:21.004423 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970001d5-26db-40e7-94b2-86e995392f21 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:21.005216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970001d5-26db-40e7-94b2-86e995392f21 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:21.005522 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970001d5-26db-40e7-94b2-86e995392f21 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:21.007127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970001d5-26db-40e7-94b2-86e995392f21 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:21.012648 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-2419e3c7-cd1d-4494-bf20-d32628acb113 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:21.013825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2419e3c7-cd1d-4494-bf20-d32628acb113 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:21.027788 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 403/812] 10.222.0.15 () {56 vars in 1133 bytes} [Tue Sep 13 01:06:20 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:21.028622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970001d5-26db-40e7-94b2-86e995392f21 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:21.071115 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:21.088405 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:21.098918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 410/813] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:20 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:21.098918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2419e3c7-cd1d-4494-bf20-d32628acb113 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:21.143465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970001d5-26db-40e7-94b2-86e995392f21 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:21.180892 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956282c0-86b3-47bf-9e06-d4759679996a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:21.181521 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956282c0-86b3-47bf-9e06-d4759679996a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:21.181805 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956282c0-86b3-47bf-9e06-d4759679996a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:21.182080 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956282c0-86b3-47bf-9e06-d4759679996a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:21.194105 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-956282c0-86b3-47bf-9e06-d4759679996a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:21.195797 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-956282c0-86b3-47bf-9e06-d4759679996a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:21.213378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-956282c0-86b3-47bf-9e06-d4759679996a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:21.213378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:21.213378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:21.213378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:21.213378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:21.213378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:21.213378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:21.213378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:21.213378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:21.213378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:21.213378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:21.213378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:21.213378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:21.213378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:21.213378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:21.213378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:21.214367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:21.214367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:21.214367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:21.214367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:21.214367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:21.214367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:21.214367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:21.214367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:21.214367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:21.214367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:21.214367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:21.214367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:21.214367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:21.214367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:21.214367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:21.214367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:21.214367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:21.215301 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:21.215301 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:21.215301 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:21.215301 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 404/814] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:21.216415 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956282c0-86b3-47bf-9e06-d4759679996a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:21.232374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2419e3c7-cd1d-4494-bf20-d32628acb113 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:21.271588 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ed05e1-3098-4b5b-bcf0-cd9de2d016e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:21.272747 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ed05e1-3098-4b5b-bcf0-cd9de2d016e9 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:21.273251 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ed05e1-3098-4b5b-bcf0-cd9de2d016e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:21.273728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ed05e1-3098-4b5b-bcf0-cd9de2d016e9 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:21.274562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ed05e1-3098-4b5b-bcf0-cd9de2d016e9 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:21.276197 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ed05e1-3098-4b5b-bcf0-cd9de2d016e9 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:21.286598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-01ed05e1-3098-4b5b-bcf0-cd9de2d016e9 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 01:06:21.286598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:21.286598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:21.286598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:21.286598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:21.286598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:21.286598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:21.286598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:21.286598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 01:06:21.286598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:21.286598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:21.286598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:21.286598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:21.286598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:21.286598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:21.286598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:21.286598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:21.288243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:21.288243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:21.288243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:21.288243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:21.288243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:21.288243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:21.288243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:21.288243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:21.288243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:21.288243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:21.288243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:21.288243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:21.288243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:21.288243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:21.288243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:21.288243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:21.289057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:21.289057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:21.289057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:21.289057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:21.289057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:21.289057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:21.289057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:21.289057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:21.289057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:21.289057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:21.289057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:21.289057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:21.289057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 01:06:21.289057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:21.292715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 411/815] 10.222.0.15 () {56 vars in 1129 bytes} [Tue Sep 13 01:06:21 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:21.293924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ed05e1-3098-4b5b-bcf0-cd9de2d016e9 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:21.344199 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:21.423696 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ed05e1-3098-4b5b-bcf0-cd9de2d016e9 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:21.456200 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956282c0-86b3-47bf-9e06-d4759679996a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:21.467232 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:21.467358 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9554f1-a64b-4cf3-9140-79f01dd40146 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:21.467813 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9554f1-a64b-4cf3-9140-79f01dd40146 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:21.468227 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9554f1-a64b-4cf3-9140-79f01dd40146 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:21.469912 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9554f1-a64b-4cf3-9140-79f01dd40146 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:21.470605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9554f1-a64b-4cf3-9140-79f01dd40146 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:21.470955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9554f1-a64b-4cf3-9140-79f01dd40146 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:21.472309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9554f1-a64b-4cf3-9140-79f01dd40146 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:21.486253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 412/816] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:21.487253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9554f1-a64b-4cf3-9140-79f01dd40146 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:21.499168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d262f5-b9d3-4df5-8e84-71fb06046c68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:21.499776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d262f5-b9d3-4df5-8e84-71fb06046c68 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:21.500050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d262f5-b9d3-4df5-8e84-71fb06046c68 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:21.500310 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d262f5-b9d3-4df5-8e84-71fb06046c68 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:21.569970 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d262f5-b9d3-4df5-8e84-71fb06046c68 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:21.571588 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d262f5-b9d3-4df5-8e84-71fb06046c68 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:21.591535 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9554f1-a64b-4cf3-9140-79f01dd40146 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:21.594577 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:21.628242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8f5247-b121-4644-b0d5-686fc2ef0139 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:21.628847 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8f5247-b121-4644-b0d5-686fc2ef0139 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:21.629108 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8f5247-b121-4644-b0d5-686fc2ef0139 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:21.629375 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8f5247-b121-4644-b0d5-686fc2ef0139 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:21.629980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8f5247-b121-4644-b0d5-686fc2ef0139 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:21.630275 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8f5247-b121-4644-b0d5-686fc2ef0139 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:21.631729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8f5247-b121-4644-b0d5-686fc2ef0139 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:21.640804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 413/817] 10.222.0.15 () {56 vars in 1124 bytes} [Tue Sep 13 01:06:21 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:21.640804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8f5247-b121-4644-b0d5-686fc2ef0139 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:21.677428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 405/818] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:06:21 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 462 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 01:06:21.678342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d262f5-b9d3-4df5-8e84-71fb06046c68 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:21.689662 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:21.753071 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8f5247-b121-4644-b0d5-686fc2ef0139 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:21.784480 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d262f5-b9d3-4df5-8e84-71fb06046c68 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:21.789894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6280bd88-d5a1-44ef-8386-c5832c4bce3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:21.790522 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280bd88-d5a1-44ef-8386-c5832c4bce3a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:21.790793 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280bd88-d5a1-44ef-8386-c5832c4bce3a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:21.791088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280bd88-d5a1-44ef-8386-c5832c4bce3a None admin] PATH_INFO: `/v3/services/network` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:21.791657 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6280bd88-d5a1-44ef-8386-c5832c4bce3a None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:21.793004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6280bd88-d5a1-44ef-8386-c5832c4bce3a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:21.801897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-6280bd88-d5a1-44ef-8386-c5832c4bce3a None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 01:06:21.801897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:21.801897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:21.801897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:21.801897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:21.801897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:21.801897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:21.801897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:21.801897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 01:06:21.801897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:21.801897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:21.801897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:21.801897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:21.801897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:21.801897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:21.801897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:21.801897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:21.803415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:21.803415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:21.803415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:21.803415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:21.803415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:21.803415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:21.803415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:21.803415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:21.803415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:21.803415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:21.803415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:21.803415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:21.803415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:21.803415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:21.803415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:21.803415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:21.804357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:21.804357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:21.804357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:21.804357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:21.804357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:21.804357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:21.804357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:21.804357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:21.804357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:21.804357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:21.804357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:21.804357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:21.804357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 01:06:21.804357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:21.804357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 414/819] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:21 2022] GET /identity/v3/services/network => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:21.805974 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6280bd88-d5a1-44ef-8386-c5832c4bce3a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:21.825714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a13cdf-3c9a-438e-9461-81fe16aeeb31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:21.826398 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a13cdf-3c9a-438e-9461-81fe16aeeb31 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:21.826680 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a13cdf-3c9a-438e-9461-81fe16aeeb31 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:21.826983 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a13cdf-3c9a-438e-9461-81fe16aeeb31 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:21.833687 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a13cdf-3c9a-438e-9461-81fe16aeeb31 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:21.835184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a13cdf-3c9a-438e-9461-81fe16aeeb31 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:21.839284 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:21.846770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-14a13cdf-3c9a-438e-9461-81fe16aeeb31 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:21.846770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:21.846770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:21.846770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:21.846770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:21.846770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:21.846770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:21.846770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:21.846770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:21.846770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:21.846770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:21.846770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:21.846770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:21.846770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:21.846770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:21.847721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:21.847721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:21.847721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:21.847721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:21.847721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:21.847721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:21.847721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:21.847721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:21.847721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:21.847721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:21.847721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:21.847721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:21.847721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:21.847721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:21.847721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:21.847721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:21.847721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:21.849207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:21.849207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:21.849207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:21.849207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:21.849207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:21.849207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:21.849207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:21.849207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:21.849207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 406/820] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:21.849207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a13cdf-3c9a-438e-9461-81fe16aeeb31 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:21.917687 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:21.919274 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6280bd88-d5a1-44ef-8386-c5832c4bce3a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:21.953129 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a13cdf-3c9a-438e-9461-81fe16aeeb31 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:21.955499 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c10b60b-d212-4070-95c9-3d544ca6f442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:21.956100 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c10b60b-d212-4070-95c9-3d544ca6f442 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:21.956374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c10b60b-d212-4070-95c9-3d544ca6f442 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:21.956641 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c10b60b-d212-4070-95c9-3d544ca6f442 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:21.957250 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c10b60b-d212-4070-95c9-3d544ca6f442 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:21.957544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c10b60b-d212-4070-95c9-3d544ca6f442 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:21.958882 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c10b60b-d212-4070-95c9-3d544ca6f442 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:21.968922 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 415/821] 10.222.0.15 () {56 vars in 1124 bytes} [Tue Sep 13 01:06:21 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:21.968922 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c10b60b-d212-4070-95c9-3d544ca6f442 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:21.995000 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21768258-ed26-4fb9-82f6-03d45b827f5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:21.995728 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21768258-ed26-4fb9-82f6-03d45b827f5e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:21.996015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21768258-ed26-4fb9-82f6-03d45b827f5e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:21.996290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21768258-ed26-4fb9-82f6-03d45b827f5e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:22.006908 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21768258-ed26-4fb9-82f6-03d45b827f5e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:22.008342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21768258-ed26-4fb9-82f6-03d45b827f5e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:22.025263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-21768258-ed26-4fb9-82f6-03d45b827f5e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:22.025263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:22.025263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:22.025263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:22.025263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:22.025263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:22.025263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:22.025263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:22.025263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:22.025263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:22.025263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:22.025263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:22.025263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:22.025263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:22.025263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:22.025263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:22.026193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:22.026193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:22.026193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:22.026193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:22.026193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:22.026193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:22.026193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:22.026193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:22.026193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:22.026193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:22.026193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:22.026193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:22.026193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:22.026193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:22.026193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:22.026193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:22.026193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:22.027123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:22.027123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:22.027123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:22.030950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 407/822] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:22.031762 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21768258-ed26-4fb9-82f6-03d45b827f5e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:22.070666 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:22.078713 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c10b60b-d212-4070-95c9-3d544ca6f442 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:22.114652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fc7844-f564-4b9f-a319-6a2e084d915e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:22.115292 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fc7844-f564-4b9f-a319-6a2e084d915e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:22.115599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fc7844-f564-4b9f-a319-6a2e084d915e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:22.116096 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fc7844-f564-4b9f-a319-6a2e084d915e None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:22.116713 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74fc7844-f564-4b9f-a319-6a2e084d915e None admin] RBAC: Adding query filter params (name=network) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:22.117158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74fc7844-f564-4b9f-a319-6a2e084d915e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:22.118508 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74fc7844-f564-4b9f-a319-6a2e084d915e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:22.123678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 416/823] 10.222.0.15 () {56 vars in 1112 bytes} [Tue Sep 13 01:06:21 2022] GET /identity/v3/services?name=network => generated 124 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:22.124550 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fc7844-f564-4b9f-a319-6a2e084d915e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:22.135818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21768258-ed26-4fb9-82f6-03d45b827f5e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:22.154968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:22.171637 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4fcc43-b5b4-4c93-aebc-41951621453c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:22.172248 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4fcc43-b5b4-4c93-aebc-41951621453c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:22.172525 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4fcc43-b5b4-4c93-aebc-41951621453c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:22.172792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4fcc43-b5b4-4c93-aebc-41951621453c None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:22.173497 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4fcc43-b5b4-4c93-aebc-41951621453c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:22.173796 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4fcc43-b5b4-4c93-aebc-41951621453c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:22.175307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4fcc43-b5b4-4c93-aebc-41951621453c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:22.187287 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 408/824] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:22.188328 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4fcc43-b5b4-4c93-aebc-41951621453c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:22.238583 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fc7844-f564-4b9f-a319-6a2e084d915e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:22.278807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce4dc00-835d-4234-92ad-8232be2230bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:22.279695 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce4dc00-835d-4234-92ad-8232be2230bd None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:22.280360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce4dc00-835d-4234-92ad-8232be2230bd None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:22.280653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce4dc00-835d-4234-92ad-8232be2230bd None admin] PATH_INFO: `/v3/endpoints` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:22.281593 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce4dc00-835d-4234-92ad-8232be2230bd None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:22.283615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce4dc00-835d-4234-92ad-8232be2230bd None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:22.289305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4fcc43-b5b4-4c93-aebc-41951621453c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:22.311199 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 417/825] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:06:22 2022] POST /identity/v3/endpoints => generated 346 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:06:22.312652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce4dc00-835d-4234-92ad-8232be2230bd None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:22.337339 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd737a2e-d3c0-4729-b4ac-1d12511a2908 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:22.337955 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd737a2e-d3c0-4729-b4ac-1d12511a2908 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:22.338231 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd737a2e-d3c0-4729-b4ac-1d12511a2908 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:22.338521 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd737a2e-d3c0-4729-b4ac-1d12511a2908 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:22.339234 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd737a2e-d3c0-4729-b4ac-1d12511a2908 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:22.339538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd737a2e-d3c0-4729-b4ac-1d12511a2908 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:22.341102 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd737a2e-d3c0-4729-b4ac-1d12511a2908 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:22.345258 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:22.359672 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 409/826] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:22.360495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd737a2e-d3c0-4729-b4ac-1d12511a2908 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:22.425627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:22.463585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce4dc00-835d-4234-92ad-8232be2230bd None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:22.505591 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd737a2e-d3c0-4729-b4ac-1d12511a2908 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:22.506115 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0b8057-5c96-4b06-96ca-4af660a166e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:22.506731 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0b8057-5c96-4b06-96ca-4af660a166e7 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:22.507050 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0b8057-5c96-4b06-96ca-4af660a166e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:22.507321 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0b8057-5c96-4b06-96ca-4af660a166e7 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:22.507995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0b8057-5c96-4b06-96ca-4af660a166e7 None admin] RBAC: Adding query filter params (type=network) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:22.508299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0b8057-5c96-4b06-96ca-4af660a166e7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:22.513117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0b8057-5c96-4b06-96ca-4af660a166e7 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:22.519838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 418/827] 10.222.0.15 () {56 vars in 1112 bytes} [Tue Sep 13 01:06:22 2022] GET /identity/v3/services?type=network => generated 350 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:22.519838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0b8057-5c96-4b06-96ca-4af660a166e7 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:22.550298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e40fbbd-30c0-4162-90e5-f2bb44899ea5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:22.551183 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e40fbbd-30c0-4162-90e5-f2bb44899ea5 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:22.551558 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e40fbbd-30c0-4162-90e5-f2bb44899ea5 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:22.551834 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e40fbbd-30c0-4162-90e5-f2bb44899ea5 None admin] PATH_INFO: `/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/e28a0ba45389417ba8ae92376dba87ff/roles/71b1e7c4b5af44ddbb7a517760ad0119` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:22.569154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e40fbbd-30c0-4162-90e5-f2bb44899ea5 None admin] RBAC: Authorizing `identity:create_grant(project_id=e5c1519fdae742f4a25eb7cd5476a8cb, user_id=e28a0ba45389417ba8ae92376dba87ff, role_id=71b1e7c4b5af44ddbb7a517760ad0119)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:22.570749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e40fbbd-30c0-4162-90e5-f2bb44899ea5 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:22.595026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 410/828] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:06:22 2022] PUT /identity/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/e28a0ba45389417ba8ae92376dba87ff/roles/71b1e7c4b5af44ddbb7a517760ad0119 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:06:22.596296 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e40fbbd-30c0-4162-90e5-f2bb44899ea5 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:22.636093 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:22.680119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0b8057-5c96-4b06-96ca-4af660a166e7 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:22.723247 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159f81a0-4566-4228-8783-f0aba00335a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:22.723885 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159f81a0-4566-4228-8783-f0aba00335a8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:22.724159 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159f81a0-4566-4228-8783-f0aba00335a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:22.724426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159f81a0-4566-4228-8783-f0aba00335a8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:22.732686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159f81a0-4566-4228-8783-f0aba00335a8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:22.736045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159f81a0-4566-4228-8783-f0aba00335a8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:22.747002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-159f81a0-4566-4228-8783-f0aba00335a8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:22.747002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:22.747002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:22.747002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:22.747002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:22.747002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:22.747002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:22.747002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:22.747002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:22.747002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:22.747002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:22.747002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:22.747002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:22.747002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:22.747002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:22.747940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:22.747940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:22.747940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:22.747940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:22.747940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:22.747940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:22.747940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:22.747940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:22.747940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:22.747940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:22.747940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:22.747940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:22.747940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:22.747940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:22.747940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:22.747940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:22.747940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:22.749319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:22.749319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:22.749319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:22.749319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:22.749319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:22.749319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:22.749319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:22.749319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:22.749319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 419/829] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:22.754324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159f81a0-4566-4228-8783-f0aba00335a8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:22.771460 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:22.794562 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e40fbbd-30c0-4162-90e5-f2bb44899ea5 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:22.844896 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111e6fdf-0ded-451b-9fd9-c40793b6b0fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:22.845518 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111e6fdf-0ded-451b-9fd9-c40793b6b0fc None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:22.845795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111e6fdf-0ded-451b-9fd9-c40793b6b0fc None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:22.846064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111e6fdf-0ded-451b-9fd9-c40793b6b0fc None admin] PATH_INFO: `/v3/endpoints` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:22.846815 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111e6fdf-0ded-451b-9fd9-c40793b6b0fc None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:22.848237 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111e6fdf-0ded-451b-9fd9-c40793b6b0fc None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:22.879832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 411/830] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:06:22 2022] POST /identity/v3/endpoints => generated 337 bytes in 284 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:06:22.882229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d42b0a2-add6-4ee4-abfe-9559b2e624b5 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:22.883036 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d42b0a2-add6-4ee4-abfe-9559b2e624b5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:22.883524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d42b0a2-add6-4ee4-abfe-9559b2e624b5 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:22.886198 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 412/831] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:22 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:22.892927 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159f81a0-4566-4228-8783-f0aba00335a8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:22.894694 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5c8d98-ae76-4ac5-b265-bdc1a6734b5e None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:22.895015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5c8d98-ae76-4ac5-b265-bdc1a6734b5e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:22.895292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5c8d98-ae76-4ac5-b265-bdc1a6734b5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:22.928988 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-cf5c8d98-ae76-4ac5-b265-bdc1a6734b5e None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:22.932214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cf5c8d98-ae76-4ac5-b265-bdc1a6734b5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:22.979218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835ab509-d29e-4033-b80b-40ec86ab4ae1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:22.979848 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835ab509-d29e-4033-b80b-40ec86ab4ae1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:22.980121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835ab509-d29e-4033-b80b-40ec86ab4ae1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:22.980387 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835ab509-d29e-4033-b80b-40ec86ab4ae1 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:22.981093 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835ab509-d29e-4033-b80b-40ec86ab4ae1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:22.981396 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835ab509-d29e-4033-b80b-40ec86ab4ae1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:22.982879 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835ab509-d29e-4033-b80b-40ec86ab4ae1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:22.992114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:23.012998 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 420/832] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:23.020093 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835ab509-d29e-4033-b80b-40ec86ab4ae1 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:23.072429 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 413/833] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:22 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:23.149673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835ab509-d29e-4033-b80b-40ec86ab4ae1 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:23.187404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e465c24-4849-466b-9e70-5154e2c109b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:23.188319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e465c24-4849-466b-9e70-5154e2c109b2 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:23.188594 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e465c24-4849-466b-9e70-5154e2c109b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:23.188861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e465c24-4849-466b-9e70-5154e2c109b2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:23.189576 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e465c24-4849-466b-9e70-5154e2c109b2 None admin] RBAC: Adding query filter params (role.id=66a36c858aab40688080eed6ceace3a0, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=2f2da4d2d3604d16b58dc25b8c17c49a) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:23.189876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e465c24-4849-466b-9e70-5154e2c109b2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:23.191313 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e465c24-4849-466b-9e70-5154e2c109b2 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:23.197715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 421/834] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:06:23 2022] GET /identity/v3/role_assignments?user.id=2f2da4d2d3604d16b58dc25b8c17c49a&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=66a36c858aab40688080eed6ceace3a0 => generated 608 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:23.399667 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5c8d98-ae76-4ac5-b265-bdc1a6734b5e None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:23.485289 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:23.517901 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5c8d98-ae76-4ac5-b265-bdc1a6734b5e None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:23.553191 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacabb53-9608-4bc3-a3a8-240ff6561354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:23.553819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacabb53-9608-4bc3-a3a8-240ff6561354 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:23.554095 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacabb53-9608-4bc3-a3a8-240ff6561354 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:23.554361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacabb53-9608-4bc3-a3a8-240ff6561354 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:23.565062 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacabb53-9608-4bc3-a3a8-240ff6561354 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:23.570292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-cacabb53-9608-4bc3-a3a8-240ff6561354 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:06:23.570292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:23.570292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:23.570292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:23.570292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:23.570292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:23.570292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:23.570292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:23.570292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:23.570292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:23.570292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:23.570292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:23.570292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:23.570292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:23.570292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:23.570292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:23.571229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:23.571229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:23.571229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:23.571229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:23.571229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:23.571229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:23.571229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:23.571229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:23.571229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:23.571229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:23.571229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:23.571229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:23.571229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:23.571229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:23.571229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:23.571229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:23.572059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:23.572059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:23.572059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:23.572059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:23.572059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:23.572059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:23.572059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:23.572059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:23.572059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:23.572059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:23.572059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:06:23.572059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:23.572059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 414/835] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:06:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:23.577534 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e465c24-4849-466b-9e70-5154e2c109b2 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:23.639966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:23.691076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e465c24-4849-466b-9e70-5154e2c109b2 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:23.725697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd82e154-94e5-4f70-806a-bb201faac497 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:23.726309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd82e154-94e5-4f70-806a-bb201faac497 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:23.726590 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd82e154-94e5-4f70-806a-bb201faac497 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:23.726875 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd82e154-94e5-4f70-806a-bb201faac497 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:23.727546 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd82e154-94e5-4f70-806a-bb201faac497 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:23.727848 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd82e154-94e5-4f70-806a-bb201faac497 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:23.729203 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd82e154-94e5-4f70-806a-bb201faac497 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:23.738310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 422/836] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:06:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:23.744411 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacabb53-9608-4bc3-a3a8-240ff6561354 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:23.847917 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacabb53-9608-4bc3-a3a8-240ff6561354 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:23.879527 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:23.884681 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05509eb-4ca7-4f9f-9926-06633f40779a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:23.885329 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05509eb-4ca7-4f9f-9926-06633f40779a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:23.885609 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05509eb-4ca7-4f9f-9926-06633f40779a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:23.885881 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05509eb-4ca7-4f9f-9926-06633f40779a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:23.902713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05509eb-4ca7-4f9f-9926-06633f40779a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:23.910835 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05509eb-4ca7-4f9f-9926-06633f40779a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:23.929350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-f05509eb-4ca7-4f9f-9926-06633f40779a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:23.929350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:23.929350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:23.929350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:23.929350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:23.929350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:23.929350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:23.929350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:23.929350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:23.929350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:23.929350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:23.929350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:23.929350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:23.929350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:23.929350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:23.929350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:23.930376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:23.930376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:23.930376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:23.930376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:23.930376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:23.930376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:23.930376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:23.930376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:23.930376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:23.930376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:23.930376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:23.930376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:23.930376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:23.930376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:23.930376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:23.930376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:23.930376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:23.931383 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:23.931383 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:23.931383 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:23.931383 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 415/837] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:23.937098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd82e154-94e5-4f70-806a-bb201faac497 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:24.043264 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd82e154-94e5-4f70-806a-bb201faac497 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:24.067425 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:24.079667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4521b947-4ce9-4a2c-a299-a85ac7a31a3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:24.080463 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4521b947-4ce9-4a2c-a299-a85ac7a31a3f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:24.080804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4521b947-4ce9-4a2c-a299-a85ac7a31a3f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:24.081295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4521b947-4ce9-4a2c-a299-a85ac7a31a3f None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:24.081939 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4521b947-4ce9-4a2c-a299-a85ac7a31a3f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:24.082242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4521b947-4ce9-4a2c-a299-a85ac7a31a3f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:24.083624 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4521b947-4ce9-4a2c-a299-a85ac7a31a3f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:24.095774 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 423/838] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:24.101600 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05509eb-4ca7-4f9f-9926-06633f40779a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:24.209243 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05509eb-4ca7-4f9f-9926-06633f40779a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:24.248497 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cc3bd0-64ea-4a29-be9e-9044ba21c6c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:24.249130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cc3bd0-64ea-4a29-be9e-9044ba21c6c1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:24.249411 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cc3bd0-64ea-4a29-be9e-9044ba21c6c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:24.249682 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cc3bd0-64ea-4a29-be9e-9044ba21c6c1 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:24.258138 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59cc3bd0-64ea-4a29-be9e-9044ba21c6c1 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:24.259599 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59cc3bd0-64ea-4a29-be9e-9044ba21c6c1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:24.271267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-59cc3bd0-64ea-4a29-be9e-9044ba21c6c1 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 13 01:06:24.271267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:24.271267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:24.271267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:24.271267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:24.271267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:24.271267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:24.271267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:24.271267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:24.271267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:24.271267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:24.271267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:24.271267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:24.271267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:24.271267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:24.271267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:24.273435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:24.273435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:24.273435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:24.273435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:24.273435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:24.273435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:24.273435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:24.273435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:24.273435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:24.273435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:24.273435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:24.273435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:24.273435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:24.273435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:24.273435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:24.273435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:24.273435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:24.273435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:24.275722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:24.275722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:24.275722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:24.275722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:24.275722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:24.275722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:24.275722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:24.275722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:24.275722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 13 01:06:24.275722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:24.277129 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 416/839] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:06:24 2022] GET /identity/v3/users/swift => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:24.278980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4521b947-4ce9-4a2c-a299-a85ac7a31a3f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:24.383564 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4521b947-4ce9-4a2c-a299-a85ac7a31a3f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:24.425870 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a943f5af-d2d7-4fe4-9b57-b3047ba79a79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:24.426729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a943f5af-d2d7-4fe4-9b57-b3047ba79a79 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:24.427230 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a943f5af-d2d7-4fe4-9b57-b3047ba79a79 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:24.427733 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a943f5af-d2d7-4fe4-9b57-b3047ba79a79 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:24.428688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a943f5af-d2d7-4fe4-9b57-b3047ba79a79 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:24.429219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a943f5af-d2d7-4fe4-9b57-b3047ba79a79 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:24.430833 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a943f5af-d2d7-4fe4-9b57-b3047ba79a79 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:24.448842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 424/840] 10.222.0.15 () {56 vars in 1135 bytes} [Tue Sep 13 01:06:24 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:24.454985 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cc3bd0-64ea-4a29-be9e-9044ba21c6c1 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:24.505615 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:24.563240 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cc3bd0-64ea-4a29-be9e-9044ba21c6c1 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:24.599186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5428d6-6106-4ddb-84fa-aff76ca43826 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:24.599928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5428d6-6106-4ddb-84fa-aff76ca43826 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:24.600211 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5428d6-6106-4ddb-84fa-aff76ca43826 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:24.600486 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5428d6-6106-4ddb-84fa-aff76ca43826 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:24.681677 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5428d6-6106-4ddb-84fa-aff76ca43826 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:24.683493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5428d6-6106-4ddb-84fa-aff76ca43826 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:24.729628 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:24.748720 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2371a434-1cec-49a1-a395-4cba003b8c7f None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:24.748720 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2371a434-1cec-49a1-a395-4cba003b8c7f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:24.749007 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2371a434-1cec-49a1-a395-4cba003b8c7f None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:24.750672 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 425/841] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:24.756609 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409488e9-e89a-43cb-bd8f-72df593c6a93 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:24.756893 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409488e9-e89a-43cb-bd8f-72df593c6a93 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:24.757167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409488e9-e89a-43cb-bd8f-72df593c6a93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:24.776455 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-409488e9-e89a-43cb-bd8f-72df593c6a93 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:24.777637 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-409488e9-e89a-43cb-bd8f-72df593c6a93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:24.791638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 417/842] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:06:24 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 01:06:24.803279 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5428d6-6106-4ddb-84fa-aff76ca43826 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:24.818267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:24.855708 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 426/843] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:24 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 01:06:24.914221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5428d6-6106-4ddb-84fa-aff76ca43826 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:24.952207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd6c551-7a66-4c46-a298-33a046ddecff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:24.952773 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd6c551-7a66-4c46-a298-33a046ddecff None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:24.953048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd6c551-7a66-4c46-a298-33a046ddecff None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:24.953316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd6c551-7a66-4c46-a298-33a046ddecff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:24.965181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd6c551-7a66-4c46-a298-33a046ddecff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:24.966745 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd6c551-7a66-4c46-a298-33a046ddecff None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:24.969541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:24.986952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-8dd6c551-7a66-4c46-a298-33a046ddecff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:24.986952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:24.986952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:24.986952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:24.986952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:24.986952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:24.986952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:24.986952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:24.986952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:24.986952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:24.986952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:24.986952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:24.986952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:24.986952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:24.986952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:24.986952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:24.987941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:24.987941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:24.987941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:24.987941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:24.987941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:24.987941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:24.987941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:24.987941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:24.987941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:24.987941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:24.987941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:24.987941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:24.987941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:24.987941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:24.987941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:24.987941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:24.987941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:24.988820 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:24.988820 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:24.988820 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:24.988820 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 418/844] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:24.993913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409488e9-e89a-43cb-bd8f-72df593c6a93 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:25.058899 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7198565d-b2e1-45c9-a525-eb87884e3165 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:25.059251 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7198565d-b2e1-45c9-a525-eb87884e3165 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:25.059441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7198565d-b2e1-45c9-a525-eb87884e3165 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:25.060967 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 419/845] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:06:25.067532 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ee2b07-5202-4ac5-8a99-4002af71abcb None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:25.068062 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ee2b07-5202-4ac5-8a99-4002af71abcb None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:25.069563 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ee2b07-5202-4ac5-8a99-4002af71abcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:25.096418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-60ee2b07-5202-4ac5-8a99-4002af71abcb None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:25.098112 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-60ee2b07-5202-4ac5-8a99-4002af71abcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:25.106983 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409488e9-e89a-43cb-bd8f-72df593c6a93 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:25.142651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae0632d-d4b6-4b97-9d07-894945ed9375 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:25.143340 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae0632d-d4b6-4b97-9d07-894945ed9375 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:25.143621 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae0632d-d4b6-4b97-9d07-894945ed9375 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:25.143893 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae0632d-d4b6-4b97-9d07-894945ed9375 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:25.144585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae0632d-d4b6-4b97-9d07-894945ed9375 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:25.144894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae0632d-d4b6-4b97-9d07-894945ed9375 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:25.146251 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae0632d-d4b6-4b97-9d07-894945ed9375 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:25.158382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 427/846] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:25.159230 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae0632d-d4b6-4b97-9d07-894945ed9375 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:25.178207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 420/847] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:25 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 01:06:25.179106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ee2b07-5202-4ac5-8a99-4002af71abcb None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:25.197821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:25.288491 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae0632d-d4b6-4b97-9d07-894945ed9375 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:25.307020 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ee2b07-5202-4ac5-8a99-4002af71abcb None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:25.328971 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41559150-bb21-4e83-806a-a2d316e534f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:25.329879 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41559150-bb21-4e83-806a-a2d316e534f0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:25.330154 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41559150-bb21-4e83-806a-a2d316e534f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:25.330421 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41559150-bb21-4e83-806a-a2d316e534f0 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:25.331026 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41559150-bb21-4e83-806a-a2d316e534f0 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:25.332370 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41559150-bb21-4e83-806a-a2d316e534f0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:25.333769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:25.342796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-41559150-bb21-4e83-806a-a2d316e534f0 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 01:06:25.342796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:25.342796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:25.342796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:25.342796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:25.342796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:25.342796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:25.342796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:25.342796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 01:06:25.342796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:25.342796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:25.342796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:25.342796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:25.342796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:25.342796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:25.342796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:25.342796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:25.343887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:25.343887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:25.343887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:25.343887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:25.343887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:25.343887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:25.343887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:25.343887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:25.343887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:25.343887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:25.343887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:25.343887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:25.343887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:25.343887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:25.343887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:25.343887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:25.344729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:25.344729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:25.344729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:25.344729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:25.344729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:25.344729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:25.344729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:25.344729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:25.344729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:25.344729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:25.344729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:25.344729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:25.344729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 01:06:25.344729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:25.345485 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e2704a-739e-4f5b-8e43-94ee07386544 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:25.345485 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e2704a-739e-4f5b-8e43-94ee07386544 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:25.345485 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e2704a-739e-4f5b-8e43-94ee07386544 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:25.345485 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e2704a-739e-4f5b-8e43-94ee07386544 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:25.347780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 428/848] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:06:25 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:25.349984 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41559150-bb21-4e83-806a-a2d316e534f0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:25.355802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e2704a-739e-4f5b-8e43-94ee07386544 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:25.357209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e2704a-739e-4f5b-8e43-94ee07386544 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:25.369115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-61e2704a-739e-4f5b-8e43-94ee07386544 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:25.369115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:25.369115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:25.369115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:25.369115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:25.369115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:25.369115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:25.369115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:25.369115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:25.369115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:25.369115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:25.369115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:25.369115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:25.369115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:25.369115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:25.371220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:25.371220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:25.371220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:25.371220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:25.371220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:25.371220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:25.371220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:25.371220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:25.371220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:25.371220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:25.371220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:25.371220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:25.371220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:25.371220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:25.371220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:25.371220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:25.371220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:25.373318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:25.373318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:25.373318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:25.373318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:25.373318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:25.373318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:25.373318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:25.373318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:25.373318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 421/849] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:25.380086 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e2704a-739e-4f5b-8e43-94ee07386544 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:25.463515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:25.470713 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41559150-bb21-4e83-806a-a2d316e534f0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:25.490940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e2704a-739e-4f5b-8e43-94ee07386544 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:25.511392 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b271f0-670d-425e-8cf4-18f0e914f5b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:25.512335 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b271f0-670d-425e-8cf4-18f0e914f5b7 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:25.512778 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b271f0-670d-425e-8cf4-18f0e914f5b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:25.513287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b271f0-670d-425e-8cf4-18f0e914f5b7 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:25.513941 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b271f0-670d-425e-8cf4-18f0e914f5b7 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:25.514249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b271f0-670d-425e-8cf4-18f0e914f5b7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:25.515724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b271f0-670d-425e-8cf4-18f0e914f5b7 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:25.526840 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 429/850] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:06:25 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:25.526840 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b271f0-670d-425e-8cf4-18f0e914f5b7 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:25.530661 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d566699-e974-4bdf-bfc8-28600d120f81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:25.531299 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d566699-e974-4bdf-bfc8-28600d120f81 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:25.531605 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d566699-e974-4bdf-bfc8-28600d120f81 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:25.531859 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d566699-e974-4bdf-bfc8-28600d120f81 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:25.532571 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d566699-e974-4bdf-bfc8-28600d120f81 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:25.532841 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d566699-e974-4bdf-bfc8-28600d120f81 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:25.534257 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d566699-e974-4bdf-bfc8-28600d120f81 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:25.551554 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 422/851] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:25.551554 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d566699-e974-4bdf-bfc8-28600d120f81 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:25.596130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:25.638831 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b271f0-670d-425e-8cf4-18f0e914f5b7 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:25.652376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d566699-e974-4bdf-bfc8-28600d120f81 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:25.672639 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3966c44b-db78-4dc8-9577-1a1e20f160c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:25.673258 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3966c44b-db78-4dc8-9577-1a1e20f160c8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:25.673537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3966c44b-db78-4dc8-9577-1a1e20f160c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:25.673805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3966c44b-db78-4dc8-9577-1a1e20f160c8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:25.683878 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3966c44b-db78-4dc8-9577-1a1e20f160c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:25.684012 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b050033-2d0a-4eb0-a656-25751219f278 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:25.684550 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b050033-2d0a-4eb0-a656-25751219f278 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:25.684825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b050033-2d0a-4eb0-a656-25751219f278 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:25.685090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b050033-2d0a-4eb0-a656-25751219f278 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:25.685703 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b050033-2d0a-4eb0-a656-25751219f278 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:25.686001 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b050033-2d0a-4eb0-a656-25751219f278 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:25.687366 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b050033-2d0a-4eb0-a656-25751219f278 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:25.688849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-3966c44b-db78-4dc8-9577-1a1e20f160c8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:06:25.688849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:25.688849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:25.688849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:25.688849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:25.688849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:25.688849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:25.688849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:25.688849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:25.688849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:25.688849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:25.688849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:25.688849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:25.688849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:25.688849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:25.688849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:25.689773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:25.689773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:25.689773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:25.689773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:25.689773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:25.689773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:25.689773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:25.689773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:25.689773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:25.689773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:25.689773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:25.689773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:25.689773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:25.689773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:25.689773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:25.689773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:25.690605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:25.690605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:25.690605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:25.690605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:25.690605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:25.690605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:25.690605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:25.690605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:25.690605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:25.690605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:25.690605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:06:25.690605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:25.690605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 430/852] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:06:25 2022] GET /identity/v3/roles/service => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:25.691359 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3966c44b-db78-4dc8-9577-1a1e20f160c8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:25.693828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 423/853] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:06:25 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:25.694498 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1059e19-d28f-4fa5-8525-a68da4d3b3c7 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:25.694777 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1059e19-d28f-4fa5-8525-a68da4d3b3c7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:25.695079 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1059e19-d28f-4fa5-8525-a68da4d3b3c7 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:25.696385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 424/854] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:25.697086 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1059e19-d28f-4fa5-8525-a68da4d3b3c7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:25.718278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:25.813628 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1059e19-d28f-4fa5-8525-a68da4d3b3c7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:25.814220 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3966c44b-db78-4dc8-9577-1a1e20f160c8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:25.827137 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:25.854037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cba82ad-37f2-4342-965d-03540b012ad0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:25.854656 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cba82ad-37f2-4342-965d-03540b012ad0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:25.854965 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cba82ad-37f2-4342-965d-03540b012ad0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:25.855243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cba82ad-37f2-4342-965d-03540b012ad0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:25.855644 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f600a39-9447-402e-a913-25950e7fa1f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:25.855971 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cba82ad-37f2-4342-965d-03540b012ad0 None admin] RBAC: Adding query filter params (role.id=66a36c858aab40688080eed6ceace3a0, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=e0d7d8ed4ede4ddca192fce0a497689d) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:25.856280 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cba82ad-37f2-4342-965d-03540b012ad0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:25.856936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f600a39-9447-402e-a913-25950e7fa1f4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:25.857415 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f600a39-9447-402e-a913-25950e7fa1f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:25.857749 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cba82ad-37f2-4342-965d-03540b012ad0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:25.858577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f600a39-9447-402e-a913-25950e7fa1f4 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:25.859285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f600a39-9447-402e-a913-25950e7fa1f4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:25.859712 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f600a39-9447-402e-a913-25950e7fa1f4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:25.861209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f600a39-9447-402e-a913-25950e7fa1f4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:25.865786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 431/855] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:06:25 2022] GET /identity/v3/role_assignments?user.id=e0d7d8ed4ede4ddca192fce0a497689d&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=66a36c858aab40688080eed6ceace3a0 => generated 608 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:25.867704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829e0ca2-4b87-480b-9998-44a1628234a8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:25.867992 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829e0ca2-4b87-480b-9998-44a1628234a8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:25.868267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829e0ca2-4b87-480b-9998-44a1628234a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:25.870423 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 425/856] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:06:25 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:25.876904 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f600a39-9447-402e-a913-25950e7fa1f4 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:25.892191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-829e0ca2-4b87-480b-9998-44a1628234a8 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:25.893776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-829e0ca2-4b87-480b-9998-44a1628234a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:25.998896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 432/857] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:25 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:26.006796 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f600a39-9447-402e-a913-25950e7fa1f4 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:26.045378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35e75e0-c3ca-44bf-ad64-80848116b4d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:26.046021 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35e75e0-c3ca-44bf-ad64-80848116b4d1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:26.046303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35e75e0-c3ca-44bf-ad64-80848116b4d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:26.046575 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35e75e0-c3ca-44bf-ad64-80848116b4d1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:26.058347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35e75e0-c3ca-44bf-ad64-80848116b4d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:26.059836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35e75e0-c3ca-44bf-ad64-80848116b4d1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:26.074914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-b35e75e0-c3ca-44bf-ad64-80848116b4d1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:26.074914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:26.074914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:26.074914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:26.074914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:26.074914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:26.074914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:26.074914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:26.074914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:26.074914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:26.074914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:26.074914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:26.074914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:26.074914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:26.074914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:26.074914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:26.075832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:26.075832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:26.075832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:26.075832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:26.075832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:26.075832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:26.075832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:26.075832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:26.075832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:26.075832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:26.075832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:26.075832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:26.075832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:26.075832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:26.075832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:26.075832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:26.075832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:26.076711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:26.076711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:26.076711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:26.076711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 426/858] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:26.081837 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829e0ca2-4b87-480b-9998-44a1628234a8 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:26.097351 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:26.187042 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829e0ca2-4b87-480b-9998-44a1628234a8 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:26.225033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2bf04a-e6a0-4633-a9e2-2e21c72c003f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:26.225650 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2bf04a-e6a0-4633-a9e2-2e21c72c003f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:26.225929 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2bf04a-e6a0-4633-a9e2-2e21c72c003f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:26.226201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2bf04a-e6a0-4633-a9e2-2e21c72c003f None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:26.226839 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2bf04a-e6a0-4633-a9e2-2e21c72c003f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:26.227179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2bf04a-e6a0-4633-a9e2-2e21c72c003f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:26.228529 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2bf04a-e6a0-4633-a9e2-2e21c72c003f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:26.241137 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 433/859] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:26.246380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35e75e0-c3ca-44bf-ad64-80848116b4d1 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:26.273889 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2bf04a-e6a0-4633-a9e2-2e21c72c003f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:26.274626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:26.359831 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:26.374561 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35e75e0-c3ca-44bf-ad64-80848116b4d1 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:26.400629 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2bf04a-e6a0-4633-a9e2-2e21c72c003f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:26.420651 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82337494-7f9e-437d-b9ec-8fac74984466 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:26.421332 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82337494-7f9e-437d-b9ec-8fac74984466 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:26.422209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82337494-7f9e-437d-b9ec-8fac74984466 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:26.422518 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82337494-7f9e-437d-b9ec-8fac74984466 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:26.433789 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82337494-7f9e-437d-b9ec-8fac74984466 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:26.435263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82337494-7f9e-437d-b9ec-8fac74984466 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:26.454894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-82337494-7f9e-437d-b9ec-8fac74984466 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 01:06:26.454894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:26.454894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:26.454894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:26.454894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:26.454894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:26.454894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:26.454894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:26.454894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:26.454894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:26.454894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:26.454894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:26.454894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:26.454894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:26.454894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:26.454894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:26.455876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:26.455876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:26.455876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:26.455876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:26.455876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:26.455876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:26.455876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:26.455876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:26.455876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:26.455876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:26.455876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:26.455876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:26.455876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:26.455876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:26.455876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:26.455876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:26.455876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:26.455876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:26.456799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:26.456799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:26.456799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:26.456799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:26.456799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:26.456799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:26.456799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:26.456799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:26.456799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 01:06:26.456799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:26.456799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 427/860] 10.222.0.15 () {56 vars in 1117 bytes} [Tue Sep 13 01:06:26 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:26.458205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82337494-7f9e-437d-b9ec-8fac74984466 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:26.459680 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80e00c1-b415-4069-8746-37a67c3b17bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:26.460539 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80e00c1-b415-4069-8746-37a67c3b17bb None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:26.460948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80e00c1-b415-4069-8746-37a67c3b17bb None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:26.461395 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80e00c1-b415-4069-8746-37a67c3b17bb None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:26.462106 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80e00c1-b415-4069-8746-37a67c3b17bb None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:26.463753 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80e00c1-b415-4069-8746-37a67c3b17bb None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:26.473519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-f80e00c1-b415-4069-8746-37a67c3b17bb None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 01:06:26.473519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:26.473519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:26.473519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:26.473519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:26.473519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:26.473519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:26.473519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:26.473519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 01:06:26.473519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:26.473519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:26.473519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:26.473519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:26.473519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:26.473519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:26.473519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:26.475355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:26.475355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:26.475355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:26.475355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:26.475355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:26.475355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:26.475355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:26.475355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:26.475355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:26.475355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:26.475355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:26.475355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:26.475355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:26.475355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:26.475355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:26.475355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:26.475355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:26.476398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:26.476398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:26.476398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:26.476398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:26.476398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:26.476398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:26.476398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:26.476398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:26.476398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:26.476398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:26.476398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:26.476398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:26.476398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 01:06:26.476398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:26.478603 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 434/861] 10.222.0.15 () {56 vars in 1132 bytes} [Tue Sep 13 01:06:26 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:26.484741 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80e00c1-b415-4069-8746-37a67c3b17bb None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:26.560851 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:26.583728 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82337494-7f9e-437d-b9ec-8fac74984466 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:26.611757 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80e00c1-b415-4069-8746-37a67c3b17bb None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:26.624901 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e288ebb-8c80-4a25-9f15-ee1f544571bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:26.625882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e288ebb-8c80-4a25-9f15-ee1f544571bf None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:26.626388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e288ebb-8c80-4a25-9f15-ee1f544571bf None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:26.626903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e288ebb-8c80-4a25-9f15-ee1f544571bf None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:26.627881 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e288ebb-8c80-4a25-9f15-ee1f544571bf None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:26.628437 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e288ebb-8c80-4a25-9f15-ee1f544571bf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:26.630160 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e288ebb-8c80-4a25-9f15-ee1f544571bf None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:26.650934 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 428/862] 10.222.0.15 () {56 vars in 1149 bytes} [Tue Sep 13 01:06:26 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:26.656313 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e288ebb-8c80-4a25-9f15-ee1f544571bf None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:26.660172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e2166c-b1b6-4b1f-8935-4609a0b6f3dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:26.660799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e2166c-b1b6-4b1f-8935-4609a0b6f3dc None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:26.661076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e2166c-b1b6-4b1f-8935-4609a0b6f3dc None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:26.661339 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e2166c-b1b6-4b1f-8935-4609a0b6f3dc None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:26.662494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e2166c-b1b6-4b1f-8935-4609a0b6f3dc None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:26.662494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e2166c-b1b6-4b1f-8935-4609a0b6f3dc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:26.663856 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e2166c-b1b6-4b1f-8935-4609a0b6f3dc None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:26.666279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:26.673787 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 435/863] 10.222.0.15 () {56 vars in 1126 bytes} [Tue Sep 13 01:06:26 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:26.676166 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e2166c-b1b6-4b1f-8935-4609a0b6f3dc None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:26.794753 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e288ebb-8c80-4a25-9f15-ee1f544571bf None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:26.795556 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e2166c-b1b6-4b1f-8935-4609a0b6f3dc None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:26.837795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa9002f-24c8-4ca4-b878-bc35633a272c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:26.838445 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa9002f-24c8-4ca4-b878-bc35633a272c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:26.838722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa9002f-24c8-4ca4-b878-bc35633a272c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:26.839026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa9002f-24c8-4ca4-b878-bc35633a272c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:26.852165 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:26.857693 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65eef44-6b67-41eb-aee3-4d6b31aeddb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:26.859557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65eef44-6b67-41eb-aee3-4d6b31aeddb0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:26.860388 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65eef44-6b67-41eb-aee3-4d6b31aeddb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:26.861231 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65eef44-6b67-41eb-aee3-4d6b31aeddb0 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:26.862511 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65eef44-6b67-41eb-aee3-4d6b31aeddb0 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:26.863301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65eef44-6b67-41eb-aee3-4d6b31aeddb0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:26.866110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65eef44-6b67-41eb-aee3-4d6b31aeddb0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:26.875015 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 436/864] 10.222.0.15 () {56 vars in 1126 bytes} [Tue Sep 13 01:06:26 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:26.926380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa9002f-24c8-4ca4-b878-bc35633a272c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:26.928309 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa9002f-24c8-4ca4-b878-bc35633a272c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:27.043328 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 429/865] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:06:26 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 01:06:27.049509 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65eef44-6b67-41eb-aee3-4d6b31aeddb0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:27.117204 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:27.157666 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65eef44-6b67-41eb-aee3-4d6b31aeddb0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:27.196705 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17170173-601d-41b3-b416-a4d2ed3cdb3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:27.197338 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17170173-601d-41b3-b416-a4d2ed3cdb3f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:27.197612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17170173-601d-41b3-b416-a4d2ed3cdb3f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:27.197878 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17170173-601d-41b3-b416-a4d2ed3cdb3f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:27.209330 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17170173-601d-41b3-b416-a4d2ed3cdb3f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:27.210763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17170173-601d-41b3-b416-a4d2ed3cdb3f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:27.225897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-17170173-601d-41b3-b416-a4d2ed3cdb3f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:27.225897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:27.225897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:27.225897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:27.225897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:27.225897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:27.225897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:27.225897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:27.225897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:27.225897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:27.225897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:27.225897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:27.225897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:27.225897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:27.225897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:27.225897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:27.226895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:27.226895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:27.226895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:27.226895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:27.226895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:27.226895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:27.226895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:27.226895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:27.226895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:27.226895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:27.226895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:27.226895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:27.226895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:27.226895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:27.226895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:27.226895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:27.226895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:27.227806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:27.227806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:27.227806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:27.227806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 437/866] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:27.234096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa9002f-24c8-4ca4-b878-bc35633a272c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:27.294121 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:27.354651 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa9002f-24c8-4ca4-b878-bc35633a272c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:27.391491 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9761502c-3f6c-4e24-9fe0-4450b1dd4a90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:27.392081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9761502c-3f6c-4e24-9fe0-4450b1dd4a90 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:27.392365 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9761502c-3f6c-4e24-9fe0-4450b1dd4a90 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:27.392639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9761502c-3f6c-4e24-9fe0-4450b1dd4a90 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:27.393329 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9761502c-3f6c-4e24-9fe0-4450b1dd4a90 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:27.393634 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9761502c-3f6c-4e24-9fe0-4450b1dd4a90 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:27.395022 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9761502c-3f6c-4e24-9fe0-4450b1dd4a90 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:27.407998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 430/867] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:27.418898 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17170173-601d-41b3-b416-a4d2ed3cdb3f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:27.520304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17170173-601d-41b3-b416-a4d2ed3cdb3f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:27.537206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:27.557356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521e14d0-1109-4d74-a60d-97335eef81e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:27.557985 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521e14d0-1109-4d74-a60d-97335eef81e7 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:27.558257 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521e14d0-1109-4d74-a60d-97335eef81e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:27.558521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521e14d0-1109-4d74-a60d-97335eef81e7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:27.565824 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521e14d0-1109-4d74-a60d-97335eef81e7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:27.567277 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521e14d0-1109-4d74-a60d-97335eef81e7 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:27.579064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-521e14d0-1109-4d74-a60d-97335eef81e7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:27.579064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:27.579064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:27.579064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:27.579064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:27.579064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:27.579064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:27.579064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:27.579064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:27.579064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:27.579064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:27.579064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:27.579064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:27.579064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:27.579064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:27.579952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:27.579952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:27.579952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:27.579952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:27.579952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:27.579952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:27.579952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:27.579952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:27.579952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:27.579952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:27.579952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:27.579952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:27.579952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:27.579952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:27.579952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:27.579952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:27.579952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:27.580820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:27.580820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:27.580820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:27.580820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:27.580820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:27.580820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:27.580820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:27.580820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:27.580820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 438/868] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:27.585923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9761502c-3f6c-4e24-9fe0-4450b1dd4a90 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:27.692244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9761502c-3f6c-4e24-9fe0-4450b1dd4a90 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:27.694180 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:27.727773 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd649d8-1559-4d7c-894d-fc98bda1670a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:27.728388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd649d8-1559-4d7c-894d-fc98bda1670a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:27.728662 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd649d8-1559-4d7c-894d-fc98bda1670a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:27.728931 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd649d8-1559-4d7c-894d-fc98bda1670a None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:27.729618 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd649d8-1559-4d7c-894d-fc98bda1670a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:27.729918 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd649d8-1559-4d7c-894d-fc98bda1670a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:27.731448 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd649d8-1559-4d7c-894d-fc98bda1670a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:27.745256 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 431/869] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:27.750919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521e14d0-1109-4d74-a60d-97335eef81e7 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:27.863981 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521e14d0-1109-4d74-a60d-97335eef81e7 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:27.901443 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a8ba08-fdf5-46be-a013-4c4ae633c746 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:27.902296 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a8ba08-fdf5-46be-a013-4c4ae633c746 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:27.902717 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a8ba08-fdf5-46be-a013-4c4ae633c746 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:27.903177 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a8ba08-fdf5-46be-a013-4c4ae633c746 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:27.904042 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a8ba08-fdf5-46be-a013-4c4ae633c746 None admin] RBAC: Adding query filter params (role.id=71b1e7c4b5af44ddbb7a517760ad0119, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=e28a0ba45389417ba8ae92376dba87ff) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:27.904509 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a8ba08-fdf5-46be-a013-4c4ae633c746 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:27.906051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a8ba08-fdf5-46be-a013-4c4ae633c746 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:27.913290 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 439/870] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:06:27 2022] GET /identity/v3/role_assignments?user.id=e28a0ba45389417ba8ae92376dba87ff&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=71b1e7c4b5af44ddbb7a517760ad0119 => generated 608 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:27.917747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e8cae8-3064-4dc0-a2f5-afe7f5a6ed18 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:27.917747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e8cae8-3064-4dc0-a2f5-afe7f5a6ed18 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:27.917747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e8cae8-3064-4dc0-a2f5-afe7f5a6ed18 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:27.917747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 432/871] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:27.924699 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5e0692-fe50-4d48-ac5b-10290ee0357a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:27.924998 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5e0692-fe50-4d48-ac5b-10290ee0357a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:27.925402 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5e0692-fe50-4d48-ac5b-10290ee0357a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:27.946347 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-2d5e0692-fe50-4d48-ac5b-10290ee0357a None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:27.947514 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2d5e0692-fe50-4d48-ac5b-10290ee0357a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:27.987379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:28.035297 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 440/872] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:27 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:28.253433 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2fef56-a639-4b65-a69a-eee3ece15601 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:28.253784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2fef56-a639-4b65-a69a-eee3ece15601 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:28.253978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2fef56-a639-4b65-a69a-eee3ece15601 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:28.255580 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 433/873] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:28.261063 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35052e9-0efb-424c-9e6e-874b1b33e955 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:28.261358 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35052e9-0efb-424c-9e6e-874b1b33e955 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:28.261636 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35052e9-0efb-424c-9e6e-874b1b33e955 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:28.280558 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b35052e9-0efb-424c-9e6e-874b1b33e955 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:28.281733 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b35052e9-0efb-424c-9e6e-874b1b33e955 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:28.311845 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2fef56-a639-4b65-a69a-eee3ece15601 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:28.356311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 441/874] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:28 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:28.440919 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2fef56-a639-4b65-a69a-eee3ece15601 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:28.476618 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46024e71-1a5a-45c9-9cf7-a1fac5713e35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:28.477594 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46024e71-1a5a-45c9-9cf7-a1fac5713e35 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:28.478093 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46024e71-1a5a-45c9-9cf7-a1fac5713e35 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:28.478605 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46024e71-1a5a-45c9-9cf7-a1fac5713e35 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:28.479541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46024e71-1a5a-45c9-9cf7-a1fac5713e35 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:28.481155 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46024e71-1a5a-45c9-9cf7-a1fac5713e35 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:28.499937 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 434/875] 10.222.0.15 () {60 vars in 1141 bytes} [Tue Sep 13 01:06:28 2022] POST /identity/v3/services => generated 251 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:06:28.640128 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35052e9-0efb-424c-9e6e-874b1b33e955 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:28.743728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:28.747386 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35052e9-0efb-424c-9e6e-874b1b33e955 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:28.789200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2289ad97-6e80-49e6-a664-b642d2bfd2d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:28.789840 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2289ad97-6e80-49e6-a664-b642d2bfd2d2 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:28.790115 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2289ad97-6e80-49e6-a664-b642d2bfd2d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:28.790382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2289ad97-6e80-49e6-a664-b642d2bfd2d2 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:28.790996 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2289ad97-6e80-49e6-a664-b642d2bfd2d2 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:28.792339 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2289ad97-6e80-49e6-a664-b642d2bfd2d2 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:28.801149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-2289ad97-6e80-49e6-a664-b642d2bfd2d2 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 01:06:28.801149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:28.801149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:28.801149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:28.801149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:28.801149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:28.801149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:28.801149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:28.801149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 01:06:28.801149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:28.801149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:28.801149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:28.801149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:28.801149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:28.801149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:28.801149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:28.801149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:28.802135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:28.802135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:28.802135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:28.802135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:28.802135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:28.802135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:28.802135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:28.802135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:28.802135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:28.802135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:28.802135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:28.802135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:28.802135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:28.802135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:28.802135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:28.802135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:28.803065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:28.803065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:28.803065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:28.803065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:28.803065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:28.803065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:28.803065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:28.803065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:28.803065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:28.803065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:28.803065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:28.803065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:28.803065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 01:06:28.803065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:28.803065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 442/876] 10.222.0.15 () {56 vars in 1126 bytes} [Tue Sep 13 01:06:28 2022] GET /identity/v3/services/object-store => generated 93 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:28.808273 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46024e71-1a5a-45c9-9cf7-a1fac5713e35 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:28.834842 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:28.920608 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46024e71-1a5a-45c9-9cf7-a1fac5713e35 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:28.955745 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddae110-d1f8-4d2e-b4ba-879c77a3f993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:28.956590 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddae110-d1f8-4d2e-b4ba-879c77a3f993 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:28.957018 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddae110-d1f8-4d2e-b4ba-879c77a3f993 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:28.957437 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddae110-d1f8-4d2e-b4ba-879c77a3f993 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:28.958230 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddae110-d1f8-4d2e-b4ba-879c77a3f993 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:28.958680 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddae110-d1f8-4d2e-b4ba-879c77a3f993 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:28.960721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddae110-d1f8-4d2e-b4ba-879c77a3f993 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:28.969903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 435/877] 10.222.0.15 () {56 vars in 1122 bytes} [Tue Sep 13 01:06:28 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:28.975880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2289ad97-6e80-49e6-a664-b642d2bfd2d2 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:29.083418 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2289ad97-6e80-49e6-a664-b642d2bfd2d2 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:29.119464 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19754bf3-16b4-4f8b-8109-ad1c43cd48fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:29.120456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19754bf3-16b4-4f8b-8109-ad1c43cd48fc None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:29.120968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19754bf3-16b4-4f8b-8109-ad1c43cd48fc None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:29.121483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19754bf3-16b4-4f8b-8109-ad1c43cd48fc None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:29.122363 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19754bf3-16b4-4f8b-8109-ad1c43cd48fc None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:29.122901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19754bf3-16b4-4f8b-8109-ad1c43cd48fc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:29.124532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19754bf3-16b4-4f8b-8109-ad1c43cd48fc None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:29.130562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 443/878] 10.222.0.15 () {56 vars in 1122 bytes} [Tue Sep 13 01:06:28 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:29.253570 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875e04aa-30cf-43b2-bd04-f83fd2c6a067 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:29.253971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875e04aa-30cf-43b2-bd04-f83fd2c6a067 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:29.254150 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875e04aa-30cf-43b2-bd04-f83fd2c6a067 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:29.255909 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 436/879] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:29.261734 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39a14d-2a65-466b-83bb-ced2c7c150f8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:29.262045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39a14d-2a65-466b-83bb-ced2c7c150f8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:29.262283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39a14d-2a65-466b-83bb-ced2c7c150f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:29.284745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8d39a14d-2a65-466b-83bb-ced2c7c150f8 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:29.285947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8d39a14d-2a65-466b-83bb-ced2c7c150f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:29.309310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:29.362152 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 444/880] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:29 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:29.640016 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875e04aa-30cf-43b2-bd04-f83fd2c6a067 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:29.728860 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:29.745474 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875e04aa-30cf-43b2-bd04-f83fd2c6a067 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:29.779699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f24267-689d-4217-8057-b5664b86d08f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:29.780304 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f24267-689d-4217-8057-b5664b86d08f None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:29.780577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f24267-689d-4217-8057-b5664b86d08f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:29.780904 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f24267-689d-4217-8057-b5664b86d08f None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:29.781548 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f24267-689d-4217-8057-b5664b86d08f None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:29.782910 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f24267-689d-4217-8057-b5664b86d08f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:29.800611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 437/881] 10.222.0.15 () {60 vars in 1141 bytes} [Tue Sep 13 01:06:29 2022] POST /identity/v3/services => generated 269 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:06:30.148485 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc42ba8-04f5-4b78-8d14-7854b2d0cf4f None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:30.149039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc42ba8-04f5-4b78-8d14-7854b2d0cf4f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:30.149039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc42ba8-04f5-4b78-8d14-7854b2d0cf4f None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:30.150581 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 445/882] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:30.155991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b377b2e-6790-4f6c-b231-adf5c19bf21d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:30.156396 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b377b2e-6790-4f6c-b231-adf5c19bf21d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:30.156584 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b377b2e-6790-4f6c-b231-adf5c19bf21d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:30.175307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7b377b2e-6790-4f6c-b231-adf5c19bf21d None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:30.176500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7b377b2e-6790-4f6c-b231-adf5c19bf21d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:30.248967 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 438/883] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:30 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:30.520315 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc42ba8-04f5-4b78-8d14-7854b2d0cf4f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:30.622751 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc42ba8-04f5-4b78-8d14-7854b2d0cf4f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:30.660829 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313060bc-e25b-4a7b-8a05-0de206772f48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:30.661510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313060bc-e25b-4a7b-8a05-0de206772f48 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:30.661729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313060bc-e25b-4a7b-8a05-0de206772f48 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:30.662084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313060bc-e25b-4a7b-8a05-0de206772f48 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:30.672418 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313060bc-e25b-4a7b-8a05-0de206772f48 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:30.677116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-313060bc-e25b-4a7b-8a05-0de206772f48 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 01:06:30.677116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:30.677116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:30.677116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:30.677116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:30.677116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:30.677116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:30.677116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:30.677116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:30.677116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:30.677116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:30.677116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:30.677116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:30.677116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:30.677116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:30.677116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:30.678563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:30.678563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:30.678563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:30.678563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:30.678563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:30.678563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:30.678563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:30.678563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:30.678563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:30.678563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:30.678563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:30.678563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:30.678563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:30.678563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:30.678563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:30.678563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:30.679432 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:30.679432 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:30.679432 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:30.679432 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:30.679432 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:30.679432 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:30.679432 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:30.679432 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:30.679432 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:30.679432 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:30.679432 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 01:06:30.679432 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:30.679432 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 446/884] 10.222.0.15 () {56 vars in 1120 bytes} [Tue Sep 13 01:06:30 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:30.684680 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b377b2e-6790-4f6c-b231-adf5c19bf21d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:30.761781 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:30.792978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b377b2e-6790-4f6c-b231-adf5c19bf21d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:30.828263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65488534-5005-44b4-8433-d144df30b004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:30.828872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65488534-5005-44b4-8433-d144df30b004 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:30.829147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65488534-5005-44b4-8433-d144df30b004 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:30.829411 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65488534-5005-44b4-8433-d144df30b004 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:30.830061 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65488534-5005-44b4-8433-d144df30b004 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:30.830358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65488534-5005-44b4-8433-d144df30b004 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:30.831355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a2f6b1-419a-4eee-8c9d-df83b1c30fc7 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:30.831732 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65488534-5005-44b4-8433-d144df30b004 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:30.832080 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a2f6b1-419a-4eee-8c9d-df83b1c30fc7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:30.832543 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a2f6b1-419a-4eee-8c9d-df83b1c30fc7 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:30.838710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 447/885] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:30 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:30.840213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 439/886] 10.222.0.15 () {56 vars in 1115 bytes} [Tue Sep 13 01:06:30 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:30.840817 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68340a30-3207-4900-b15a-014258b3538f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:30.841221 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68340a30-3207-4900-b15a-014258b3538f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:30.841559 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68340a30-3207-4900-b15a-014258b3538f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:30.847800 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65488534-5005-44b4-8433-d144df30b004 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:30.860876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-68340a30-3207-4900-b15a-014258b3538f None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:30.862118 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-68340a30-3207-4900-b15a-014258b3538f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:30.875230 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:30.944815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 448/887] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:30 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 01:06:30.968885 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65488534-5005-44b4-8433-d144df30b004 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:31.005493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d70575-accf-455b-9015-a0df831d671e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:31.006195 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d70575-accf-455b-9015-a0df831d671e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:31.006471 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d70575-accf-455b-9015-a0df831d671e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:31.006740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d70575-accf-455b-9015-a0df831d671e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:31.008655 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:31.018697 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d70575-accf-455b-9015-a0df831d671e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:31.020293 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d70575-accf-455b-9015-a0df831d671e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:31.034956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-b3d70575-accf-455b-9015-a0df831d671e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:31.034956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:31.034956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:31.034956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:31.034956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:31.034956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:31.034956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:31.034956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:31.034956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:31.034956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:31.034956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:31.034956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:31.034956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:31.034956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:31.034956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:31.034956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:31.035906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:31.035906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:31.035906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:31.035906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:31.035906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:31.035906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:31.035906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:31.035906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:31.035906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:31.035906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:31.035906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:31.035906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:31.035906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:31.035906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:31.035906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:31.035906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:31.035906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:31.037300 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:31.037300 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:31.037300 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:31.037300 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 440/888] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:31.042960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68340a30-3207-4900-b15a-014258b3538f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:31.145252 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68340a30-3207-4900-b15a-014258b3538f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:31.181107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5418ec64-8fb8-4f67-be56-e0d1a7c3c213 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:31.181857 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5418ec64-8fb8-4f67-be56-e0d1a7c3c213 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:31.182015 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5418ec64-8fb8-4f67-be56-e0d1a7c3c213 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:31.182233 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5418ec64-8fb8-4f67-be56-e0d1a7c3c213 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:31.182879 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5418ec64-8fb8-4f67-be56-e0d1a7c3c213 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:31.183197 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5418ec64-8fb8-4f67-be56-e0d1a7c3c213 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:31.184603 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5418ec64-8fb8-4f67-be56-e0d1a7c3c213 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:31.196811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 449/889] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:31.202191 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d70575-accf-455b-9015-a0df831d671e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:31.219161 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5418ec64-8fb8-4f67-be56-e0d1a7c3c213 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:31.223056 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:31.333165 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5418ec64-8fb8-4f67-be56-e0d1a7c3c213 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:31.337342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d70575-accf-455b-9015-a0df831d671e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:31.371924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38499852-ea72-4b0c-8199-d0bc3f5374c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:31.372942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38499852-ea72-4b0c-8199-d0bc3f5374c5 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:31.373392 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38499852-ea72-4b0c-8199-d0bc3f5374c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:31.373920 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38499852-ea72-4b0c-8199-d0bc3f5374c5 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:31.374633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38499852-ea72-4b0c-8199-d0bc3f5374c5 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:31.376447 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38499852-ea72-4b0c-8199-d0bc3f5374c5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:31.383218 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9e76fb-681c-436d-add5-26fbfd3058e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:31.383848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9e76fb-681c-436d-add5-26fbfd3058e1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:31.384125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9e76fb-681c-436d-add5-26fbfd3058e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:31.384398 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9e76fb-681c-436d-add5-26fbfd3058e1 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:31.387555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-38499852-ea72-4b0c-8199-d0bc3f5374c5 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 01:06:31.387555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:31.387555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:31.387555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:31.387555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:31.387555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:31.387555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:31.387555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:31.387555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 01:06:31.387555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:31.387555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:31.387555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:31.387555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:31.387555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:31.387555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:31.387555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:31.387555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:31.388503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:31.388503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:31.388503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:31.388503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:31.388503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:31.388503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:31.388503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:31.388503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:31.388503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:31.388503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:31.388503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:31.388503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:31.388503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:31.388503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:31.388503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:31.388503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:31.389317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:31.389317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:31.389317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:31.389317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:31.389317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:31.389317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:31.389317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:31.389317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:31.389317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:31.389317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:31.389317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:31.389317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:31.389317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 01:06:31.389317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:31.392753 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9e76fb-681c-436d-add5-26fbfd3058e1 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:31.396826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9e76fb-681c-436d-add5-26fbfd3058e1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:31.397190 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 450/890] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:06:31 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:31.398589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38499852-ea72-4b0c-8199-d0bc3f5374c5 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:31.406742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-6a9e76fb-681c-436d-add5-26fbfd3058e1 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 01:06:31.406742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:31.406742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:31.406742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:31.406742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:31.406742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:31.406742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:31.406742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:31.406742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:31.406742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:31.406742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:31.406742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:31.406742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:31.406742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:31.406742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:31.406742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:31.407724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:31.407724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:31.407724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:31.407724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:31.407724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:31.407724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:31.407724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:31.407724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:31.407724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:31.407724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:31.407724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:31.407724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:31.407724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:31.407724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:31.407724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:31.407724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:31.407724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:31.407724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:31.408969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:31.408969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:31.408969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:31.408969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:31.408969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:31.408969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:31.408969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:31.408969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:31.408969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 01:06:31.408969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:31.413346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 441/891] 10.222.0.15 () {56 vars in 1117 bytes} [Tue Sep 13 01:06:31 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:31.414231 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9e76fb-681c-436d-add5-26fbfd3058e1 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:31.472728 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:31.479807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:31.518315 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38499852-ea72-4b0c-8199-d0bc3f5374c5 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:31.528092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9e76fb-681c-436d-add5-26fbfd3058e1 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:31.554135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de184e44-75ae-46fa-83df-3a3e49d16254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:31.554738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de184e44-75ae-46fa-83df-3a3e49d16254 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:31.555045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de184e44-75ae-46fa-83df-3a3e49d16254 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:31.555317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de184e44-75ae-46fa-83df-3a3e49d16254 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:31.555934 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de184e44-75ae-46fa-83df-3a3e49d16254 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:31.556232 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de184e44-75ae-46fa-83df-3a3e49d16254 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:31.557579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de184e44-75ae-46fa-83df-3a3e49d16254 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:31.566801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 451/892] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:06:31 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:31.566801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2df4b96-1bff-46f1-932f-264fb83b2123 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:31.566801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2df4b96-1bff-46f1-932f-264fb83b2123 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:31.566801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2df4b96-1bff-46f1-932f-264fb83b2123 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:31.566801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 452/893] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:31.570554 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2df4b96-1bff-46f1-932f-264fb83b2123 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:31.571141 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb82de6-02fa-4f28-9914-a55bca4acdf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:31.571755 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb82de6-02fa-4f28-9914-a55bca4acdf4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:31.572028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb82de6-02fa-4f28-9914-a55bca4acdf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:31.572295 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb82de6-02fa-4f28-9914-a55bca4acdf4 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:31.573024 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb82de6-02fa-4f28-9914-a55bca4acdf4 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:31.573514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb82de6-02fa-4f28-9914-a55bca4acdf4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:31.575938 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb82de6-02fa-4f28-9914-a55bca4acdf4 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:31.602682 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 442/894] 10.222.0.15 () {56 vars in 1149 bytes} [Tue Sep 13 01:06:31 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:31.602682 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587d7085-2c7c-41bf-9729-9a1c179aa955 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:31.602682 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587d7085-2c7c-41bf-9729-9a1c179aa955 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:31.602682 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587d7085-2c7c-41bf-9729-9a1c179aa955 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:31.621300 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-587d7085-2c7c-41bf-9729-9a1c179aa955 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:31.622667 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-587d7085-2c7c-41bf-9729-9a1c179aa955 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:31.680033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2df4b96-1bff-46f1-932f-264fb83b2123 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:31.706994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 443/895] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:31 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:31.707890 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587d7085-2c7c-41bf-9729-9a1c179aa955 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:31.718677 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:31.725849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff208fc3-8054-458c-a2a4-5be50403f983 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:31.727254 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff208fc3-8054-458c-a2a4-5be50403f983 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:31.727987 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff208fc3-8054-458c-a2a4-5be50403f983 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:31.728439 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff208fc3-8054-458c-a2a4-5be50403f983 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:31.729496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff208fc3-8054-458c-a2a4-5be50403f983 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:31.730377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff208fc3-8054-458c-a2a4-5be50403f983 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:31.731946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff208fc3-8054-458c-a2a4-5be50403f983 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:31.739225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 453/896] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:06:31 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:31.744604 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff208fc3-8054-458c-a2a4-5be50403f983 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:31.750419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:31.836346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587d7085-2c7c-41bf-9729-9a1c179aa955 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:31.856110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff208fc3-8054-458c-a2a4-5be50403f983 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:31.884564 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c586cf4-56b1-4305-886d-5c2690c335a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:31.885700 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c586cf4-56b1-4305-886d-5c2690c335a0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:31.886633 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c586cf4-56b1-4305-886d-5c2690c335a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:31.887343 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c586cf4-56b1-4305-886d-5c2690c335a0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:31.894730 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c23371c-4a41-464a-9589-4f7f3d1342d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:31.895479 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c23371c-4a41-464a-9589-4f7f3d1342d4 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:31.895849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c23371c-4a41-464a-9589-4f7f3d1342d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:31.896358 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c23371c-4a41-464a-9589-4f7f3d1342d4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:31.897145 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c23371c-4a41-464a-9589-4f7f3d1342d4 None admin] RBAC: Adding query filter params (interface=public, service_id=f612f4d588c646b392d0182b9f1ce783, region_id=RegionOne) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:31.897551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c23371c-4a41-464a-9589-4f7f3d1342d4 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:31.899014 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c23371c-4a41-464a-9589-4f7f3d1342d4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:31.909079 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 454/897] 10.222.0.15 () {56 vars in 1251 bytes} [Tue Sep 13 01:06:31 2022] GET /identity/v3/endpoints?service_id=f612f4d588c646b392d0182b9f1ce783&interface=public®ion_id=RegionOne => generated 194 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:31.910044 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c23371c-4a41-464a-9589-4f7f3d1342d4 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:31.978679 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:31.983110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c586cf4-56b1-4305-886d-5c2690c335a0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:31.984565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c586cf4-56b1-4305-886d-5c2690c335a0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:32.018869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:32.025699 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c23371c-4a41-464a-9589-4f7f3d1342d4 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:32.061431 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88b9a01-87d1-4deb-b429-ae07914a1166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:32.062027 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88b9a01-87d1-4deb-b429-ae07914a1166 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:32.062303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88b9a01-87d1-4deb-b429-ae07914a1166 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:32.062573 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88b9a01-87d1-4deb-b429-ae07914a1166 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:32.063459 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88b9a01-87d1-4deb-b429-ae07914a1166 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:32.064805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88b9a01-87d1-4deb-b429-ae07914a1166 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:32.070466 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 455/898] 10.222.0.15 () {56 vars in 1087 bytes} [Tue Sep 13 01:06:31 2022] GET /identity/v3/services => generated 1268 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:06:32.071363 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88b9a01-87d1-4deb-b429-ae07914a1166 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:32.103179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 444/899] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:06:31 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 01:06:32.104374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243ae1d8-e9e6-4f6f-b174-1036f40004a7 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:32.104657 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243ae1d8-e9e6-4f6f-b174-1036f40004a7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:32.104927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243ae1d8-e9e6-4f6f-b174-1036f40004a7 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:32.106401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 445/900] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:32.111288 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243ae1d8-e9e6-4f6f-b174-1036f40004a7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:32.188117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88b9a01-87d1-4deb-b429-ae07914a1166 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:32.213745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:32.226523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb7836f-740a-44e9-be06-d60dadfc5b75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:32.227177 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb7836f-740a-44e9-be06-d60dadfc5b75 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:32.227451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb7836f-740a-44e9-be06-d60dadfc5b75 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:32.227807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb7836f-740a-44e9-be06-d60dadfc5b75 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:32.228461 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb7836f-740a-44e9-be06-d60dadfc5b75 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:32.228627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243ae1d8-e9e6-4f6f-b174-1036f40004a7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:32.229887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb7836f-740a-44e9-be06-d60dadfc5b75 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:32.248063 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 456/901] 10.222.0.15 () {60 vars in 1141 bytes} [Tue Sep 13 01:06:32 2022] POST /identity/v3/services => generated 241 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:06:32.249083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9098a8-b967-4545-acb5-fc170d96a2e4 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:32.249368 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9098a8-b967-4545-acb5-fc170d96a2e4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:32.249701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9098a8-b967-4545-acb5-fc170d96a2e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:32.270174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7a9098a8-b967-4545-acb5-fc170d96a2e4 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:32.270812 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:32.272101 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7a9098a8-b967-4545-acb5-fc170d96a2e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:32.278970 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28939587-d9b0-4520-8e07-032413f244d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:32.279964 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28939587-d9b0-4520-8e07-032413f244d3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:32.280418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28939587-d9b0-4520-8e07-032413f244d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:32.280861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28939587-d9b0-4520-8e07-032413f244d3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:32.295195 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28939587-d9b0-4520-8e07-032413f244d3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:32.296973 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28939587-d9b0-4520-8e07-032413f244d3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:32.315194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-28939587-d9b0-4520-8e07-032413f244d3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:32.315194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:32.315194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:32.315194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:32.315194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:32.315194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:32.315194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:32.315194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:32.315194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:32.315194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:32.315194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:32.315194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:32.315194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:32.315194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:32.315194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:32.315194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:32.316214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:32.316214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:32.316214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:32.316214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:32.316214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:32.316214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:32.316214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:32.316214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:32.316214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:32.316214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:32.316214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:32.316214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:32.316214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:32.316214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:32.316214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:32.316214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:32.316214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:32.317085 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:32.317085 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:32.317085 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:32.319497 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 446/902] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:32.325636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28939587-d9b0-4520-8e07-032413f244d3 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:32.365686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 457/903] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:32 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:32.447924 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28939587-d9b0-4520-8e07-032413f244d3 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:32.487250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb58f35-fce2-4626-9a9f-c06abb40412e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:32.487788 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb58f35-fce2-4626-9a9f-c06abb40412e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:32.488070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb58f35-fce2-4626-9a9f-c06abb40412e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:32.488343 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb58f35-fce2-4626-9a9f-c06abb40412e None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:32.488985 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb58f35-fce2-4626-9a9f-c06abb40412e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:32.489286 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb58f35-fce2-4626-9a9f-c06abb40412e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:32.490717 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb58f35-fce2-4626-9a9f-c06abb40412e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:32.507141 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 447/904] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:32.510120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9098a8-b967-4545-acb5-fc170d96a2e4 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:32.613632 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9098a8-b967-4545-acb5-fc170d96a2e4 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:32.628566 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:32.644581 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb58f35-fce2-4626-9a9f-c06abb40412e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:32.655310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9dc81e-5eb9-45a8-9a32-c14e6b62e43c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:32.655959 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9dc81e-5eb9-45a8-9a32-c14e6b62e43c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:32.656237 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9dc81e-5eb9-45a8-9a32-c14e6b62e43c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:32.656505 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9dc81e-5eb9-45a8-9a32-c14e6b62e43c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:32.666075 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9dc81e-5eb9-45a8-9a32-c14e6b62e43c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:32.668039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9dc81e-5eb9-45a8-9a32-c14e6b62e43c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:32.685134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-9e9dc81e-5eb9-45a8-9a32-c14e6b62e43c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:32.685134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:32.685134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:32.685134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:32.685134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:32.685134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:32.685134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:32.685134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:32.685134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:32.685134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:32.685134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:32.685134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:32.685134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:32.685134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:32.685134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:32.686025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:32.686025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:32.686025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:32.686025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:32.686025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:32.686025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:32.686025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:32.686025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:32.686025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:32.686025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:32.686025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:32.686025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:32.686025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:32.686025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:32.686025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:32.686025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:32.686025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:32.686945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:32.686945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:32.686945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:32.686945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:32.686945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:32.686945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:32.686945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:32.686945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:32.690958 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 458/905] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:32.692516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9dc81e-5eb9-45a8-9a32-c14e6b62e43c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:32.697375 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:32.770146 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb58f35-fce2-4626-9a9f-c06abb40412e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:32.807035 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9dc81e-5eb9-45a8-9a32-c14e6b62e43c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:32.809099 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce085fe-aba6-42e3-a3cf-1f4e1d6d784e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:32.809744 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce085fe-aba6-42e3-a3cf-1f4e1d6d784e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:32.810022 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce085fe-aba6-42e3-a3cf-1f4e1d6d784e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:32.810294 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce085fe-aba6-42e3-a3cf-1f4e1d6d784e None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:32.811513 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce085fe-aba6-42e3-a3cf-1f4e1d6d784e None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:32.812380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce085fe-aba6-42e3-a3cf-1f4e1d6d784e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:32.822638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-9ce085fe-aba6-42e3-a3cf-1f4e1d6d784e None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 01:06:32.822638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:32.822638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:32.822638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:32.822638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:32.822638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:32.822638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:32.822638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:32.822638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 01:06:32.822638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:32.822638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:32.822638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:32.822638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:32.822638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:32.822638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:32.822638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:32.823652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:32.823652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:32.823652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:32.823652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:32.823652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:32.823652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:32.823652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:32.823652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:32.823652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:32.823652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:32.823652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:32.823652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:32.823652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:32.823652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:32.823652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:32.823652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:32.823652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:32.824517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:32.824517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:32.824517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:32.824517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:32.824517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:32.824517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:32.824517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:32.824517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:32.824517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:32.824517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:32.824517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:32.824517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:32.824517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 01:06:32.824517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:32.824517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 448/906] 10.222.0.15 () {56 vars in 1132 bytes} [Tue Sep 13 01:06:32 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:32.831234 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce085fe-aba6-42e3-a3cf-1f4e1d6d784e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:32.851382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1db141-36b3-459f-a766-731e77e53f26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:32.852366 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1db141-36b3-459f-a766-731e77e53f26 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:32.852883 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1db141-36b3-459f-a766-731e77e53f26 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:32.853345 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1db141-36b3-459f-a766-731e77e53f26 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:32.854209 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1db141-36b3-459f-a766-731e77e53f26 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:32.854686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1db141-36b3-459f-a766-731e77e53f26 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:32.856319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1db141-36b3-459f-a766-731e77e53f26 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:32.870134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 459/907] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:32.876992 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1db141-36b3-459f-a766-731e77e53f26 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:32.914035 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:32.969477 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce085fe-aba6-42e3-a3cf-1f4e1d6d784e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:32.978605 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:32.990711 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1db141-36b3-459f-a766-731e77e53f26 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:33.014540 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06424a4-deb7-46fd-940b-75f15b217160 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:33.015964 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06424a4-deb7-46fd-940b-75f15b217160 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:33.016366 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06424a4-deb7-46fd-940b-75f15b217160 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:33.016745 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06424a4-deb7-46fd-940b-75f15b217160 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:33.017916 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06424a4-deb7-46fd-940b-75f15b217160 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:33.018336 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06424a4-deb7-46fd-940b-75f15b217160 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:33.020132 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06424a4-deb7-46fd-940b-75f15b217160 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:33.032444 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 449/908] 10.222.0.15 () {56 vars in 1126 bytes} [Tue Sep 13 01:06:32 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:33.034009 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06424a4-deb7-46fd-940b-75f15b217160 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:33.038937 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f418ffe2-1b7d-45e7-a612-e144fc1c6a5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:33.040736 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f418ffe2-1b7d-45e7-a612-e144fc1c6a5c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:33.041209 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f418ffe2-1b7d-45e7-a612-e144fc1c6a5c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:33.041587 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f418ffe2-1b7d-45e7-a612-e144fc1c6a5c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:33.042741 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f418ffe2-1b7d-45e7-a612-e144fc1c6a5c None admin] RBAC: Adding query filter params (role.id=11f2e919cae541428f8c564b7b67eea0, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=e28a0ba45389417ba8ae92376dba87ff) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:33.043199 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f418ffe2-1b7d-45e7-a612-e144fc1c6a5c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:33.044935 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f418ffe2-1b7d-45e7-a612-e144fc1c6a5c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:33.057949 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 460/909] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:06:32 2022] GET /identity/v3/role_assignments?user.id=e28a0ba45389417ba8ae92376dba87ff&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=11f2e919cae541428f8c564b7b67eea0 => generated 259 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:33.152947 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06424a4-deb7-46fd-940b-75f15b217160 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:33.192816 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6610c62-005c-4c60-821f-321a98d0e527 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:33.193975 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6610c62-005c-4c60-821f-321a98d0e527 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:33.194251 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6610c62-005c-4c60-821f-321a98d0e527 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:33.194520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6610c62-005c-4c60-821f-321a98d0e527 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:33.195244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6610c62-005c-4c60-821f-321a98d0e527 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:33.195551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6610c62-005c-4c60-821f-321a98d0e527 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:33.196928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6610c62-005c-4c60-821f-321a98d0e527 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:33.202887 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 450/910] 10.222.0.15 () {56 vars in 1126 bytes} [Tue Sep 13 01:06:33 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:33.211980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f418ffe2-1b7d-45e7-a612-e144fc1c6a5c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:33.320543 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:33.327200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f418ffe2-1b7d-45e7-a612-e144fc1c6a5c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:33.363530 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f316c5-9292-4125-b2c6-891f7a6d91f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:33.364142 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f316c5-9292-4125-b2c6-891f7a6d91f7 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:33.364418 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f316c5-9292-4125-b2c6-891f7a6d91f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:33.364692 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f316c5-9292-4125-b2c6-891f7a6d91f7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:33.365450 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f316c5-9292-4125-b2c6-891f7a6d91f7 None admin] RBAC: Adding query filter params (interface=public, service_id=5a9ab8e190c44d55a07603c816070203, region_id=RegionOne) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:33.365760 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f316c5-9292-4125-b2c6-891f7a6d91f7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:33.367170 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f316c5-9292-4125-b2c6-891f7a6d91f7 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:33.372800 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 461/911] 10.222.0.15 () {56 vars in 1251 bytes} [Tue Sep 13 01:06:33 2022] GET /identity/v3/endpoints?service_id=5a9ab8e190c44d55a07603c816070203&interface=public®ion_id=RegionOne => generated 194 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:33.378468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6610c62-005c-4c60-821f-321a98d0e527 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:33.428728 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:33.504304 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6610c62-005c-4c60-821f-321a98d0e527 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:33.540936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31538d8b-4946-420d-9b2f-8934a24845fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:33.541914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31538d8b-4946-420d-9b2f-8934a24845fe None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:33.542390 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31538d8b-4946-420d-9b2f-8934a24845fe None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:33.542886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31538d8b-4946-420d-9b2f-8934a24845fe None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:33.543697 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31538d8b-4946-420d-9b2f-8934a24845fe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:33.545247 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31538d8b-4946-420d-9b2f-8934a24845fe None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:33.552011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 451/912] 10.222.0.15 () {56 vars in 1087 bytes} [Tue Sep 13 01:06:33 2022] GET /identity/v3/services => generated 1497 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:06:34.312198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c666ab-ad97-4992-8ba4-7a3db28cddc1 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:34.312748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c666ab-ad97-4992-8ba4-7a3db28cddc1 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:34.312748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c666ab-ad97-4992-8ba4-7a3db28cddc1 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:34.314566 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 462/913] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:34.320141 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc5a447-b367-4986-b372-af820b78d5a0 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:34.320434 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc5a447-b367-4986-b372-af820b78d5a0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:34.320713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc5a447-b367-4986-b372-af820b78d5a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:34.340864 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8cc5a447-b367-4986-b372-af820b78d5a0 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:34.342121 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8cc5a447-b367-4986-b372-af820b78d5a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:34.426360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 452/914] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:34 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:34.452141 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fde571-29a0-4830-a8ce-47de39ee40b2 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:34.452451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fde571-29a0-4830-a8ce-47de39ee40b2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:34.452693 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fde571-29a0-4830-a8ce-47de39ee40b2 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:34.454358 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 463/915] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:34.460656 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0a9d2a-7d3c-4396-87d3-b299738321ca None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:34.460967 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0a9d2a-7d3c-4396-87d3-b299738321ca None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:34.461208 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0a9d2a-7d3c-4396-87d3-b299738321ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:34.479653 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-2b0a9d2a-7d3c-4396-87d3-b299738321ca None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:34.480840 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2b0a9d2a-7d3c-4396-87d3-b299738321ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:34.504029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:34.548427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 453/916] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:34 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:34.703986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fde571-29a0-4830-a8ce-47de39ee40b2 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:34.816408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0a9d2a-7d3c-4396-87d3-b299738321ca None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:34.828805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fde571-29a0-4830-a8ce-47de39ee40b2 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:34.866820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410aa060-629e-47fe-8a89-071d80cf198b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:34.867943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410aa060-629e-47fe-8a89-071d80cf198b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:34.868549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410aa060-629e-47fe-8a89-071d80cf198b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:34.868973 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410aa060-629e-47fe-8a89-071d80cf198b None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:34.869714 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410aa060-629e-47fe-8a89-071d80cf198b None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:34.871262 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410aa060-629e-47fe-8a89-071d80cf198b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:34.881516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-410aa060-629e-47fe-8a89-071d80cf198b None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 01:06:34.881516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:34.881516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:34.881516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:34.881516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:34.881516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:34.881516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:34.881516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:34.881516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 01:06:34.881516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:34.881516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:34.881516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:34.881516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:34.881516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:34.881516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:34.881516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:34.881516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:34.882724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:34.882724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:34.882724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:34.882724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:34.882724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:34.882724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:34.882724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:34.882724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:34.882724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:34.882724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:34.882724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:34.882724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:34.882724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:34.882724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:34.882724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:34.882724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:34.883786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:34.883786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:34.883786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:34.883786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:34.883786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:34.883786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:34.883786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:34.883786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:34.883786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:34.883786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:34.883786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:34.883786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:34.883786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 01:06:34.883786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:34.886241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 464/917] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:06:34 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:34.892165 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410aa060-629e-47fe-8a89-071d80cf198b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:34.923984 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:34.926184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0a9d2a-7d3c-4396-87d3-b299738321ca None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:34.966023 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e24b36-7884-4ba9-b6b7-facdcd9ce431 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:34.966951 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e24b36-7884-4ba9-b6b7-facdcd9ce431 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:34.967237 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e24b36-7884-4ba9-b6b7-facdcd9ce431 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:34.967535 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e24b36-7884-4ba9-b6b7-facdcd9ce431 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:34.968101 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e24b36-7884-4ba9-b6b7-facdcd9ce431 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:34.969942 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e24b36-7884-4ba9-b6b7-facdcd9ce431 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:34.979055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-25e24b36-7884-4ba9-b6b7-facdcd9ce431 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 01:06:34.979055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:34.979055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:34.979055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:34.979055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:34.979055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:34.979055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:34.979055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:34.979055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 01:06:34.979055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:34.979055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:34.979055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:34.979055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:34.979055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:34.979055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:34.979055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:34.979055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:34.980017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:34.980017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:34.980017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:34.980017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:34.980017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:34.980017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:34.980017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:34.980017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:34.980017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:34.980017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:34.980017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:34.980017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:34.980017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:34.980017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:34.980017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:34.980017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:34.980867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:34.980867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:34.980867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:34.980867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:34.980867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:34.980867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:34.980867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:34.980867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:34.980867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:34.980867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:34.980867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:34.980867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:34.980867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 01:06:34.980867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:34.980867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 454/918] 10.222.0.15 () {56 vars in 1126 bytes} [Tue Sep 13 01:06:34 2022] GET /identity/v3/services/object-store => generated 93 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:34.987439 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e24b36-7884-4ba9-b6b7-facdcd9ce431 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:35.022364 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:35.024847 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410aa060-629e-47fe-8a89-071d80cf198b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:35.064390 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67601167-78ea-40a2-a1a5-7402f8d4471a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:35.065342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67601167-78ea-40a2-a1a5-7402f8d4471a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:35.065857 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67601167-78ea-40a2-a1a5-7402f8d4471a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:35.066344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67601167-78ea-40a2-a1a5-7402f8d4471a None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:35.067219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67601167-78ea-40a2-a1a5-7402f8d4471a None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:35.067888 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67601167-78ea-40a2-a1a5-7402f8d4471a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:35.070500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67601167-78ea-40a2-a1a5-7402f8d4471a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:35.076452 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 465/919] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:06:34 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:35.085723 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67601167-78ea-40a2-a1a5-7402f8d4471a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:35.129256 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e24b36-7884-4ba9-b6b7-facdcd9ce431 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:35.167687 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df29c4e7-26e0-4462-b09e-7c63612e4a04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:35.168773 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df29c4e7-26e0-4462-b09e-7c63612e4a04 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:35.169295 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df29c4e7-26e0-4462-b09e-7c63612e4a04 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:35.169766 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df29c4e7-26e0-4462-b09e-7c63612e4a04 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:35.170729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df29c4e7-26e0-4462-b09e-7c63612e4a04 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:35.171249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df29c4e7-26e0-4462-b09e-7c63612e4a04 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:35.172982 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df29c4e7-26e0-4462-b09e-7c63612e4a04 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:35.183325 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 455/920] 10.222.0.15 () {56 vars in 1122 bytes} [Tue Sep 13 01:06:34 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:35.184567 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:35.185051 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df29c4e7-26e0-4462-b09e-7c63612e4a04 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:35.195354 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67601167-78ea-40a2-a1a5-7402f8d4471a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:35.236877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353e6f90-ddc2-4ace-a6b2-c2a9868b0c97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:35.237514 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353e6f90-ddc2-4ace-a6b2-c2a9868b0c97 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:35.237792 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353e6f90-ddc2-4ace-a6b2-c2a9868b0c97 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:35.238058 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353e6f90-ddc2-4ace-a6b2-c2a9868b0c97 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:35.238681 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-353e6f90-ddc2-4ace-a6b2-c2a9868b0c97 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:35.239006 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-353e6f90-ddc2-4ace-a6b2-c2a9868b0c97 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:35.240626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-353e6f90-ddc2-4ace-a6b2-c2a9868b0c97 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:35.250955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 466/921] 10.222.0.15 () {56 vars in 1114 bytes} [Tue Sep 13 01:06:35 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:35.253600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353e6f90-ddc2-4ace-a6b2-c2a9868b0c97 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:35.293332 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:35.310624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df29c4e7-26e0-4462-b09e-7c63612e4a04 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:35.353253 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80b87ca-1a14-4988-9937-3636b3e4e4c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:35.353976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80b87ca-1a14-4988-9937-3636b3e4e4c7 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:35.354261 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80b87ca-1a14-4988-9937-3636b3e4e4c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:35.354535 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80b87ca-1a14-4988-9937-3636b3e4e4c7 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:35.355495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80b87ca-1a14-4988-9937-3636b3e4e4c7 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:35.355674 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80b87ca-1a14-4988-9937-3636b3e4e4c7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:35.357022 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80b87ca-1a14-4988-9937-3636b3e4e4c7 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:35.368136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 456/922] 10.222.0.15 () {56 vars in 1122 bytes} [Tue Sep 13 01:06:35 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:35.369326 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80b87ca-1a14-4988-9937-3636b3e4e4c7 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:35.372357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353e6f90-ddc2-4ace-a6b2-c2a9868b0c97 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:35.411553 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d72c6f4-a3aa-4b2f-8c45-d900e3a0b74b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:35.412902 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d72c6f4-a3aa-4b2f-8c45-d900e3a0b74b None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:35.413161 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d72c6f4-a3aa-4b2f-8c45-d900e3a0b74b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:35.413498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d72c6f4-a3aa-4b2f-8c45-d900e3a0b74b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:35.414220 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d72c6f4-a3aa-4b2f-8c45-d900e3a0b74b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:35.415599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d72c6f4-a3aa-4b2f-8c45-d900e3a0b74b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:35.437521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 467/923] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:06:35 2022] POST /identity/v3/endpoints => generated 346 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:06:35.439431 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e16b42-738a-401b-bc94-eaf5f3dc1362 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:35.439896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e16b42-738a-401b-bc94-eaf5f3dc1362 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:35.440172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e16b42-738a-401b-bc94-eaf5f3dc1362 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:35.441703 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 468/924] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:35.447970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd99ef3a-a791-4977-9848-6441ed8fc03a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:35.448310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd99ef3a-a791-4977-9848-6441ed8fc03a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:35.448521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd99ef3a-a791-4977-9848-6441ed8fc03a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:35.469212 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-dd99ef3a-a791-4977-9848-6441ed8fc03a None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:35.470445 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-dd99ef3a-a791-4977-9848-6441ed8fc03a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:35.475796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:35.499692 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80b87ca-1a14-4988-9937-3636b3e4e4c7 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:35.555146 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66295ca-2c3e-4aff-9da4-952ecd7723bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:35.556090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66295ca-2c3e-4aff-9da4-952ecd7723bb None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:35.556457 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66295ca-2c3e-4aff-9da4-952ecd7723bb None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:35.556844 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66295ca-2c3e-4aff-9da4-952ecd7723bb None admin] PATH_INFO: `/v3/endpoints` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:35.557634 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66295ca-2c3e-4aff-9da4-952ecd7723bb None admin] RBAC: Adding query filter params (interface=public, service_id=12f35fe97fcf4d6a828249b14c3e6aae, region_id=RegionOne) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:35.558015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66295ca-2c3e-4aff-9da4-952ecd7723bb None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:35.559188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 469/925] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:35 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 01:06:35.563853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66295ca-2c3e-4aff-9da4-952ecd7723bb None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:35.570990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 457/926] 10.222.0.15 () {56 vars in 1251 bytes} [Tue Sep 13 01:06:35 2022] GET /identity/v3/endpoints?service_id=12f35fe97fcf4d6a828249b14c3e6aae&interface=public®ion_id=RegionOne => generated 194 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:35.576434 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd99ef3a-a791-4977-9848-6441ed8fc03a None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:35.695529 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd99ef3a-a791-4977-9848-6441ed8fc03a None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:35.732031 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f27cab9-5583-4a3f-9bf0-bf4d432c28ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:35.732651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f27cab9-5583-4a3f-9bf0-bf4d432c28ea None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:35.732930 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f27cab9-5583-4a3f-9bf0-bf4d432c28ea None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:35.733204 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f27cab9-5583-4a3f-9bf0-bf4d432c28ea None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:35.733807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f27cab9-5583-4a3f-9bf0-bf4d432c28ea None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:35.735188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f27cab9-5583-4a3f-9bf0-bf4d432c28ea None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:35.742243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 470/927] 10.222.0.15 () {56 vars in 1087 bytes} [Tue Sep 13 01:06:35 2022] GET /identity/v3/services => generated 1497 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:06:35.781566 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0247ffe5-1545-47df-95a5-f5451731bd32 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:35.781766 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0247ffe5-1545-47df-95a5-f5451731bd32 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:35.782032 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0247ffe5-1545-47df-95a5-f5451731bd32 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:35.783729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 458/928] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:35.789548 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaddc535-2ba3-46cf-b723-d1aa029be40d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:35.789834 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaddc535-2ba3-46cf-b723-d1aa029be40d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:35.790111 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaddc535-2ba3-46cf-b723-d1aa029be40d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:35.798736 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:35.811734 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-aaddc535-2ba3-46cf-b723-d1aa029be40d None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:35.813216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-aaddc535-2ba3-46cf-b723-d1aa029be40d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:35.836426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0247ffe5-1545-47df-95a5-f5451731bd32 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:35.850480 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:35.905202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 471/929] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:35 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:35.955738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0247ffe5-1545-47df-95a5-f5451731bd32 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:35.993144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfca3569-393c-43ce-a0d0-01efb018cd4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:35.994195 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfca3569-393c-43ce-a0d0-01efb018cd4e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:35.994716 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfca3569-393c-43ce-a0d0-01efb018cd4e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:35.995232 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfca3569-393c-43ce-a0d0-01efb018cd4e None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:36.006975 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfca3569-393c-43ce-a0d0-01efb018cd4e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:36.012147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-dfca3569-393c-43ce-a0d0-01efb018cd4e None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 01:06:36.012147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:36.012147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:36.012147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:36.012147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:36.012147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:36.012147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:36.012147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:36.012147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:36.012147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:36.012147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:36.012147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:36.012147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:36.012147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:36.012147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:36.012147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:36.013075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:36.013075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:36.013075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:36.013075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:36.013075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:36.013075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:36.013075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:36.013075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:36.013075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:36.013075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:36.013075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:36.013075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:36.013075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:36.013075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:36.013075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:36.013075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:36.013903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:36.013903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:36.013903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:36.013903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:36.013903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:36.013903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:36.013903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:36.013903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:36.013903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:36.013903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:36.013903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 01:06:36.013903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:36.019133 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 459/930] 10.222.0.15 () {56 vars in 1120 bytes} [Tue Sep 13 01:06:35 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:36.023670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaddc535-2ba3-46cf-b723-d1aa029be40d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:36.134241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaddc535-2ba3-46cf-b723-d1aa029be40d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:36.169132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fc7173-2995-40d7-a2ef-141c2f44d7c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:36.169744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fc7173-2995-40d7-a2ef-141c2f44d7c7 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:36.170021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fc7173-2995-40d7-a2ef-141c2f44d7c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:36.170293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fc7173-2995-40d7-a2ef-141c2f44d7c7 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:36.170976 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fc7173-2995-40d7-a2ef-141c2f44d7c7 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:36.171279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fc7173-2995-40d7-a2ef-141c2f44d7c7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:36.172776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fc7173-2995-40d7-a2ef-141c2f44d7c7 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:36.173208 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfca3569-393c-43ce-a0d0-01efb018cd4e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:36.174224 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:36.183878 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 472/931] 10.222.0.15 () {56 vars in 1115 bytes} [Tue Sep 13 01:06:36 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:36.190181 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fc7173-2995-40d7-a2ef-141c2f44d7c7 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:36.250940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:36.290263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfca3569-393c-43ce-a0d0-01efb018cd4e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:36.332897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fc7173-2995-40d7-a2ef-141c2f44d7c7 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:36.334160 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6143a2d9-0b90-4b46-97ed-60735fa8dedf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:36.336293 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6143a2d9-0b90-4b46-97ed-60735fa8dedf None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:36.336802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6143a2d9-0b90-4b46-97ed-60735fa8dedf None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:36.337279 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6143a2d9-0b90-4b46-97ed-60735fa8dedf None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:36.338092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6143a2d9-0b90-4b46-97ed-60735fa8dedf None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:36.339712 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6143a2d9-0b90-4b46-97ed-60735fa8dedf None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:36.352636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-6143a2d9-0b90-4b46-97ed-60735fa8dedf None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 01:06:36.352636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:36.352636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:36.352636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:36.352636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:36.352636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:36.352636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:36.352636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:36.352636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 01:06:36.352636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:36.352636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:36.352636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:36.352636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:36.352636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:36.352636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:36.352636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:36.353592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:36.353592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:36.353592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:36.353592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:36.353592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:36.353592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:36.353592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:36.353592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:36.353592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:36.353592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:36.353592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:36.353592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:36.353592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:36.353592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:36.353592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:36.353592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:36.353592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:36.354468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:36.354468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:36.354468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:36.354468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:36.354468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:36.354468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:36.354468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:36.354468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:36.354468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:36.354468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:36.354468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:36.354468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:36.354468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 01:06:36.354468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:36.358785 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 460/932] 10.222.0.15 () {56 vars in 1132 bytes} [Tue Sep 13 01:06:36 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:36.359744 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6143a2d9-0b90-4b46-97ed-60735fa8dedf None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:36.382373 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c73af4-592c-490f-9e9c-a156ad538bfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:36.383823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c73af4-592c-490f-9e9c-a156ad538bfd None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:36.385020 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c73af4-592c-490f-9e9c-a156ad538bfd None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:36.385442 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c73af4-592c-490f-9e9c-a156ad538bfd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:36.406940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c73af4-592c-490f-9e9c-a156ad538bfd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:36.409453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c73af4-592c-490f-9e9c-a156ad538bfd None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:36.434198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-40c73af4-592c-490f-9e9c-a156ad538bfd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:36.434198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:36.434198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:36.434198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:36.434198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:36.434198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:36.434198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:36.434198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:36.434198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:36.434198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:36.434198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:36.434198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:36.434198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:36.434198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:36.434198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:36.434198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:36.435557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:36.435557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:36.435557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:36.435557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:36.435557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:36.435557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:36.435557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:36.435557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:36.435557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:36.435557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:36.435557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:36.435557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:36.435557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:36.435557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:36.435557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:36.435557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:36.435557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:36.436783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:36.436783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:36.436783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:36.440744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 473/933] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:36.441797 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c73af4-592c-490f-9e9c-a156ad538bfd None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:36.495442 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:36.500853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6143a2d9-0b90-4b46-97ed-60735fa8dedf None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:36.550904 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfd7e01-07c8-45fe-b093-c0e834557828 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:36.552035 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfd7e01-07c8-45fe-b093-c0e834557828 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:36.552950 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c73af4-592c-490f-9e9c-a156ad538bfd None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:36.553225 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfd7e01-07c8-45fe-b093-c0e834557828 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:36.553673 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfd7e01-07c8-45fe-b093-c0e834557828 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:36.554471 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfd7e01-07c8-45fe-b093-c0e834557828 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:36.555182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfd7e01-07c8-45fe-b093-c0e834557828 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:36.557650 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfd7e01-07c8-45fe-b093-c0e834557828 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:36.558931 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:36.565200 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 461/934] 10.222.0.15 () {56 vars in 1126 bytes} [Tue Sep 13 01:06:36 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:36.570939 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfd7e01-07c8-45fe-b093-c0e834557828 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:36.595726 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b05ba97-8c22-49c8-bbff-20d2dd706047 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:36.596350 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b05ba97-8c22-49c8-bbff-20d2dd706047 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:36.596650 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b05ba97-8c22-49c8-bbff-20d2dd706047 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:36.596894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b05ba97-8c22-49c8-bbff-20d2dd706047 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:36.597540 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b05ba97-8c22-49c8-bbff-20d2dd706047 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:36.597846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b05ba97-8c22-49c8-bbff-20d2dd706047 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:36.599890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b05ba97-8c22-49c8-bbff-20d2dd706047 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:36.622374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 474/935] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:36.623307 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b05ba97-8c22-49c8-bbff-20d2dd706047 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:36.709505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfd7e01-07c8-45fe-b093-c0e834557828 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:36.748516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b05ba97-8c22-49c8-bbff-20d2dd706047 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:36.754346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b540ce-8ec8-4258-b1a2-43eeec3a1b64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:36.755303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b540ce-8ec8-4258-b1a2-43eeec3a1b64 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:36.755739 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b540ce-8ec8-4258-b1a2-43eeec3a1b64 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:36.756106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b540ce-8ec8-4258-b1a2-43eeec3a1b64 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:36.757149 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b540ce-8ec8-4258-b1a2-43eeec3a1b64 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:36.757673 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b540ce-8ec8-4258-b1a2-43eeec3a1b64 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:36.759971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b540ce-8ec8-4258-b1a2-43eeec3a1b64 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:36.761262 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:36.774757 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 462/936] 10.222.0.15 () {56 vars in 1126 bytes} [Tue Sep 13 01:06:36 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:36.776528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b540ce-8ec8-4258-b1a2-43eeec3a1b64 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:36.795252 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1ce95a-26b4-4442-a675-330fc3ed73aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:36.796444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1ce95a-26b4-4442-a675-330fc3ed73aa None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:36.797288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1ce95a-26b4-4442-a675-330fc3ed73aa None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:36.797778 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1ce95a-26b4-4442-a675-330fc3ed73aa None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:36.808587 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1ce95a-26b4-4442-a675-330fc3ed73aa None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:36.811149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1ce95a-26b4-4442-a675-330fc3ed73aa None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:36.825918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-aa1ce95a-26b4-4442-a675-330fc3ed73aa None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 01:06:36.825918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:36.825918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:36.825918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:36.825918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:36.825918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:36.825918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:36.825918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:36.825918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:36.825918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:36.825918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:36.825918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:36.825918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:36.825918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:36.825918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:36.825918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:36.827117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:36.827117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:36.827117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:36.827117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:36.827117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:36.827117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:36.827117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:36.827117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:36.827117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:36.827117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:36.827117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:36.827117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:36.827117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:36.827117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:36.827117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:36.827117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:36.827117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:36.827117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:36.828244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:36.828244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:36.828244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:36.828244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:36.828244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:36.828244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:36.828244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:36.828244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:36.828244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 01:06:36.828244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:36.828244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 475/937] 10.222.0.15 () {56 vars in 1117 bytes} [Tue Sep 13 01:06:36 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:36.832589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1ce95a-26b4-4442-a675-330fc3ed73aa None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:36.853549 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:36.905010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b540ce-8ec8-4258-b1a2-43eeec3a1b64 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:36.939596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1ce95a-26b4-4442-a675-330fc3ed73aa None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:36.950086 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c7d058-e7ec-4399-8ee0-79ff717e00a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:36.950902 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c7d058-e7ec-4399-8ee0-79ff717e00a3 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:36.951627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c7d058-e7ec-4399-8ee0-79ff717e00a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:36.952213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c7d058-e7ec-4399-8ee0-79ff717e00a3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:36.953137 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c7d058-e7ec-4399-8ee0-79ff717e00a3 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:36.954806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c7d058-e7ec-4399-8ee0-79ff717e00a3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:36.978500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebf1e67-4275-497d-bef7-01096f77e533 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:36.979318 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebf1e67-4275-497d-bef7-01096f77e533 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:36.979698 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebf1e67-4275-497d-bef7-01096f77e533 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:36.980075 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebf1e67-4275-497d-bef7-01096f77e533 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:36.981020 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebf1e67-4275-497d-bef7-01096f77e533 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:36.981428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebf1e67-4275-497d-bef7-01096f77e533 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:36.982967 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 463/938] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:06:36 2022] POST /identity/v3/endpoints => generated 347 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:06:36.986799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebf1e67-4275-497d-bef7-01096f77e533 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:37.010415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 476/939] 10.222.0.15 () {56 vars in 1149 bytes} [Tue Sep 13 01:06:36 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:37.016101 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c7d058-e7ec-4399-8ee0-79ff717e00a3 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:37.146871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c7d058-e7ec-4399-8ee0-79ff717e00a3 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:37.163534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:37.190600 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a0baa9-efcc-4a23-b0bf-78489fe81735 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:37.191519 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a0baa9-efcc-4a23-b0bf-78489fe81735 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:37.191946 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a0baa9-efcc-4a23-b0bf-78489fe81735 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:37.192366 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a0baa9-efcc-4a23-b0bf-78489fe81735 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:37.204373 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a0baa9-efcc-4a23-b0bf-78489fe81735 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:37.206046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a0baa9-efcc-4a23-b0bf-78489fe81735 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:37.221249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-f4a0baa9-efcc-4a23-b0bf-78489fe81735 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:37.221249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:37.221249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:37.221249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:37.221249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:37.221249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:37.221249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:37.221249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:37.221249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:37.221249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:37.221249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:37.221249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:37.221249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:37.221249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:37.221249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:37.221249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:37.222387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:37.222387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:37.222387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:37.222387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:37.222387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:37.222387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:37.222387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:37.222387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:37.222387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:37.222387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:37.222387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:37.222387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:37.222387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:37.222387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:37.222387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:37.222387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:37.222387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:37.223545 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:37.223545 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:37.223545 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:37.225885 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 464/940] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:37.232528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebf1e67-4275-497d-bef7-01096f77e533 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:37.262577 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:37.346197 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebf1e67-4275-497d-bef7-01096f77e533 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:37.383360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4eecb42-0f00-43d2-a936-efaa6b1a201c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:37.384710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4eecb42-0f00-43d2-a936-efaa6b1a201c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:37.385249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4eecb42-0f00-43d2-a936-efaa6b1a201c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:37.385736 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4eecb42-0f00-43d2-a936-efaa6b1a201c None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:37.386606 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4eecb42-0f00-43d2-a936-efaa6b1a201c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:37.387107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4eecb42-0f00-43d2-a936-efaa6b1a201c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:37.388700 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4eecb42-0f00-43d2-a936-efaa6b1a201c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:37.411054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 477/941] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:37.418330 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a0baa9-efcc-4a23-b0bf-78489fe81735 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:37.526686 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a0baa9-efcc-4a23-b0bf-78489fe81735 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:37.564474 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e7cfb2-4269-4187-8c0a-78be43a7d463 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:37.565535 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e7cfb2-4269-4187-8c0a-78be43a7d463 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:37.565817 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e7cfb2-4269-4187-8c0a-78be43a7d463 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:37.566093 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e7cfb2-4269-4187-8c0a-78be43a7d463 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:37.576066 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e7cfb2-4269-4187-8c0a-78be43a7d463 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:37.577796 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e7cfb2-4269-4187-8c0a-78be43a7d463 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:37.589746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-81e7cfb2-4269-4187-8c0a-78be43a7d463 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:37.589746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:37.589746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:37.589746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:37.589746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:37.589746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:37.589746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:37.589746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:37.589746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:37.589746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:37.589746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:37.589746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:37.589746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:37.589746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:37.589746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:37.590616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:37.590616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:37.590616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:37.590616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:37.590616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:37.590616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:37.590616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:37.590616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:37.590616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:37.590616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:37.590616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:37.590616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:37.590616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:37.590616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:37.590616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:37.590616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:37.590616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:37.591528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:37.591528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:37.591528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:37.591528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:37.591528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:37.591528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:37.591528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:37.591528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:37.593813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 465/942] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:37.599851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4eecb42-0f00-43d2-a936-efaa6b1a201c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:37.705408 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:37.721595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4eecb42-0f00-43d2-a936-efaa6b1a201c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:37.757633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a905d03-c966-4609-89b6-72403dff809e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:37.758326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a905d03-c966-4609-89b6-72403dff809e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:37.758607 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a905d03-c966-4609-89b6-72403dff809e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:37.758899 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a905d03-c966-4609-89b6-72403dff809e None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:37.759594 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a905d03-c966-4609-89b6-72403dff809e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:37.759901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a905d03-c966-4609-89b6-72403dff809e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:37.760068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb9dbe5-9620-4c75-a585-32c7ff4bd872 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:37.760318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb9dbe5-9620-4c75-a585-32c7ff4bd872 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:37.760598 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb9dbe5-9620-4c75-a585-32c7ff4bd872 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:37.761318 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a905d03-c966-4609-89b6-72403dff809e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:37.762189 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 466/943] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:37.767337 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc5af7f-2001-4de5-a35c-3007a075f636 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:37.767616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc5af7f-2001-4de5-a35c-3007a075f636 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:37.767889 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc5af7f-2001-4de5-a35c-3007a075f636 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:37.774987 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 478/944] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:37.783896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a905d03-c966-4609-89b6-72403dff809e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:37.789139 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-5cc5af7f-2001-4de5-a35c-3007a075f636 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:37.790340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5cc5af7f-2001-4de5-a35c-3007a075f636 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:37.812139 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:37.873276 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 467/945] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:37 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 01:06:37.897916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a905d03-c966-4609-89b6-72403dff809e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:37.933485 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15f5306-d460-4a69-a571-8942e1b18906 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:37.934160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15f5306-d460-4a69-a571-8942e1b18906 None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:37.934488 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15f5306-d460-4a69-a571-8942e1b18906 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:37.934760 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15f5306-d460-4a69-a571-8942e1b18906 None admin] PATH_INFO: `/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/e28a0ba45389417ba8ae92376dba87ff/roles/11f2e919cae541428f8c564b7b67eea0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:37.937056 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15f5306-d460-4a69-a571-8942e1b18906 None admin] RBAC: Authorizing `identity:create_grant(project_id=e5c1519fdae742f4a25eb7cd5476a8cb, user_id=e28a0ba45389417ba8ae92376dba87ff, role_id=11f2e919cae541428f8c564b7b67eea0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:37.938764 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15f5306-d460-4a69-a571-8942e1b18906 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:37.955755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 479/946] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:06:37 2022] PUT /identity/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/e28a0ba45389417ba8ae92376dba87ff/roles/11f2e919cae541428f8c564b7b67eea0 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:06:38.074962 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297797bc-fff9-48e7-99dd-ec7d73295caa None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:38.075349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297797bc-fff9-48e7-99dd-ec7d73295caa None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:38.075744 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297797bc-fff9-48e7-99dd-ec7d73295caa None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:38.077492 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 468/947] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:38.085673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9beb54e9-f0a1-4e90-b861-e6b65a175491 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:38.086191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9beb54e9-f0a1-4e90-b861-e6b65a175491 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:38.086470 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9beb54e9-f0a1-4e90-b861-e6b65a175491 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:38.099480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:38.106998 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9beb54e9-f0a1-4e90-b861-e6b65a175491 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:38.108715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9beb54e9-f0a1-4e90-b861-e6b65a175491 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:38.153084 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297797bc-fff9-48e7-99dd-ec7d73295caa None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:38.185334 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 480/948] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:38 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 01:06:38.266427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297797bc-fff9-48e7-99dd-ec7d73295caa None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:38.303786 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be5ff36-50db-4fcf-8fb4-51b539037036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:38.304632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be5ff36-50db-4fcf-8fb4-51b539037036 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:38.305140 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be5ff36-50db-4fcf-8fb4-51b539037036 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:38.305570 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be5ff36-50db-4fcf-8fb4-51b539037036 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:38.306404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be5ff36-50db-4fcf-8fb4-51b539037036 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:38.308108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be5ff36-50db-4fcf-8fb4-51b539037036 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:38.310898 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: WARNING keystone.server.flask.common [None req-0be5ff36-50db-4fcf-8fb4-51b539037036 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 13 01:06:38.331936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.sql.core [None req-0be5ff36-50db-4fcf-8fb4-51b539037036 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 13 01:06:38.331936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 13 01:06:38.331936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [parameters: {'id': '086b6a190388462fbceb809a60154104', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 13 01:06:38.331936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81404) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 13 01:06:38.339944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-0be5ff36-50db-4fcf-8fb4-51b539037036 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 086b6a190388462fbceb809a60154104.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 086b6a190388462fbceb809a60154104. Sep 13 01:06:38.339944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:38.339944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 13 01:06:38.339944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 13 01:06:38.339944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 13 01:06:38.339944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 13 01:06:38.339944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 13 01:06:38.339944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application result = self._query(query) Sep 13 01:06:38.339944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 13 01:06:38.339944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application conn.query(q) Sep 13 01:06:38.339944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 13 01:06:38.339944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 13 01:06:38.339944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 13 01:06:38.341701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application result.read() Sep 13 01:06:38.341701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 13 01:06:38.341701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 13 01:06:38.341701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 13 01:06:38.341701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 13 01:06:38.341701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 13 01:06:38.341701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 13 01:06:38.341701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 13 01:06:38.341701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 13 01:06:38.341701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 13 01:06:38.341701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:38.341701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 13 01:06:38.341701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:38.341701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:38.341701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 13 01:06:38.341701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 13 01:06:38.341701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 13 01:06:38.341701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 13 01:06:38.343157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 13 01:06:38.343157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application next(self.gen) Sep 13 01:06:38.343157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 13 01:06:38.343157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application yield resource Sep 13 01:06:38.343157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 13 01:06:38.343157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application next(self.gen) Sep 13 01:06:38.343157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 13 01:06:38.343157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self.session.rollback() Sep 13 01:06:38.343157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 13 01:06:38.343157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self.force_reraise() Sep 13 01:06:38.343157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 13 01:06:38.343157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise self.value Sep 13 01:06:38.343157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 13 01:06:38.343157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 13 01:06:38.343157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 13 01:06:38.343157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application session.commit() Sep 13 01:06:38.343157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 13 01:06:38.344038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 13 01:06:38.344038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 13 01:06:38.344038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._prepare_impl() Sep 13 01:06:38.344038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 13 01:06:38.344038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self.session.flush() Sep 13 01:06:38.344038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 13 01:06:38.344038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._flush(objects) Sep 13 01:06:38.344038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 13 01:06:38.344038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 13 01:06:38.344038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 13 01:06:38.344038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application compat.raise_( Sep 13 01:06:38.344038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 13 01:06:38.344038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception Sep 13 01:06:38.344038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 13 01:06:38.344038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application flush_context.execute() Sep 13 01:06:38.344038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 13 01:06:38.344038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rec.execute(self) Sep 13 01:06:38.344925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 13 01:06:38.344925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 13 01:06:38.344925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 13 01:06:38.344925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 13 01:06:38.344925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 13 01:06:38.344925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 13 01:06:38.344925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 13 01:06:38.344925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 13 01:06:38.344925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 13 01:06:38.344925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 13 01:06:38.344925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 13 01:06:38.344925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 13 01:06:38.344925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 13 01:06:38.344925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 13 01:06:38.344925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 13 01:06:38.345711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 13 01:06:38.345711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 13 01:06:38.345711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception Sep 13 01:06:38.345711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 13 01:06:38.345711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 13 01:06:38.345711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 13 01:06:38.345711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 13 01:06:38.345711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 13 01:06:38.345711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application result = self._query(query) Sep 13 01:06:38.345711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 13 01:06:38.345711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application conn.query(q) Sep 13 01:06:38.345711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 13 01:06:38.345711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 13 01:06:38.345711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 13 01:06:38.345711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application result.read() Sep 13 01:06:38.345711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 13 01:06:38.345711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 13 01:06:38.346577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 13 01:06:38.346577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 13 01:06:38.346577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 13 01:06:38.346577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 13 01:06:38.346577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 13 01:06:38.346577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 13 01:06:38.346577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 13 01:06:38.346577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 13 01:06:38.346577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [parameters: {'id': '086b6a190388462fbceb809a60154104', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 13 01:06:38.346577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 13 01:06:38.346577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:38.346577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:38.346577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:38.346577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:38.346577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 13 01:06:38.347401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 13 01:06:38.347401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 13 01:06:38.347401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 13 01:06:38.347401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 13 01:06:38.347401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:38.347401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:38.347401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:38.347401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:38.347401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:38.347401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:38.347401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:38.347401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:38.347401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:38.347401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:38.347401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:38.347401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:38.347401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:38.348874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:38.348874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 13 01:06:38.348874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 13 01:06:38.348874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:38.348874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:38.348874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 13 01:06:38.348874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 13 01:06:38.348874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 086b6a190388462fbceb809a60154104. Sep 13 01:06:38.348874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:38.348874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 469/949] 10.222.0.15 () {60 vars in 1140 bytes} [Tue Sep 13 01:06:38 2022] POST /identity/v3/projects => generated 262 bytes in 189 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:38.356078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9beb54e9-f0a1-4e90-b861-e6b65a175491 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:38.466877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9beb54e9-f0a1-4e90-b861-e6b65a175491 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:38.478160 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be5ff36-50db-4fcf-8fb4-51b539037036 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:38.497379 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:38.526616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be68d8a8-cb48-4797-b204-ba0dc1c1bdc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:38.528172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be68d8a8-cb48-4797-b204-ba0dc1c1bdc3 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:38.528453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be68d8a8-cb48-4797-b204-ba0dc1c1bdc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:38.528727 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be68d8a8-cb48-4797-b204-ba0dc1c1bdc3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:38.532142 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:38.539380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be68d8a8-cb48-4797-b204-ba0dc1c1bdc3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:38.541661 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be68d8a8-cb48-4797-b204-ba0dc1c1bdc3 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:38.558517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-be68d8a8-cb48-4797-b204-ba0dc1c1bdc3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:38.558517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:38.558517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:38.558517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:38.558517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:38.558517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:38.558517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:38.558517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:38.558517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:38.558517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:38.558517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:38.558517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:38.558517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:38.558517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:38.558517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:38.560218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:38.560218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:38.560218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:38.560218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:38.560218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:38.560218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:38.560218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:38.560218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:38.560218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:38.560218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:38.560218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:38.560218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:38.560218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:38.560218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:38.560218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:38.560218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:38.560218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:38.561298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:38.561298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:38.561298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:38.561298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:38.561298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:38.561298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:38.561298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:38.561298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:38.566872 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 481/950] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:38.566872 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be68d8a8-cb48-4797-b204-ba0dc1c1bdc3 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:38.618517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be5ff36-50db-4fcf-8fb4-51b539037036 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:38.659516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd84c0a0-58a1-4486-8ff9-2560f4c090a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:38.660255 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd84c0a0-58a1-4486-8ff9-2560f4c090a7 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:38.660593 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd84c0a0-58a1-4486-8ff9-2560f4c090a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:38.660925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd84c0a0-58a1-4486-8ff9-2560f4c090a7 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:38.661610 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd84c0a0-58a1-4486-8ff9-2560f4c090a7 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:38.663255 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd84c0a0-58a1-4486-8ff9-2560f4c090a7 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:38.672217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-cd84c0a0-58a1-4486-8ff9-2560f4c090a7 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 01:06:38.672217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:38.672217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:38.672217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:38.672217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:38.672217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:38.672217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:38.672217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:38.672217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 01:06:38.672217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:38.672217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:38.672217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:38.672217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:38.672217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:38.672217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:38.672217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:38.672217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:38.674284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:38.674284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:38.674284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:38.674284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:38.674284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:38.674284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:38.674284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:38.674284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:38.674284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:38.674284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:38.674284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:38.674284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:38.674284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:38.674284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:38.674284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:38.674284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:38.675352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:38.675352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:38.675352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:38.675352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:38.675352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:38.675352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:38.675352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:38.675352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:38.675352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:38.675352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:38.675352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:38.675352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:38.675352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 01:06:38.675352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:38.675352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 470/951] 10.222.0.15 () {56 vars in 1126 bytes} [Tue Sep 13 01:06:38 2022] GET /identity/v3/services/object-store => generated 93 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:38.681371 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd84c0a0-58a1-4486-8ff9-2560f4c090a7 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:38.696810 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be68d8a8-cb48-4797-b204-ba0dc1c1bdc3 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:38.744890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810d0576-ca34-4a04-98e2-3516a827a29f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:38.745510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810d0576-ca34-4a04-98e2-3516a827a29f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:38.745794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810d0576-ca34-4a04-98e2-3516a827a29f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:38.746067 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810d0576-ca34-4a04-98e2-3516a827a29f None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:38.746760 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810d0576-ca34-4a04-98e2-3516a827a29f None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:38.747597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810d0576-ca34-4a04-98e2-3516a827a29f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:38.749334 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810d0576-ca34-4a04-98e2-3516a827a29f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:38.773310 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:38.783153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 482/952] 10.222.0.15 () {56 vars in 1112 bytes} [Tue Sep 13 01:06:38 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:38.809240 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd84c0a0-58a1-4486-8ff9-2560f4c090a7 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:38.849158 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e580b8-c76f-4e68-9f1d-eb619efc0664 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:38.849774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e580b8-c76f-4e68-9f1d-eb619efc0664 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:38.850049 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e580b8-c76f-4e68-9f1d-eb619efc0664 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:38.850323 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e580b8-c76f-4e68-9f1d-eb619efc0664 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:38.851040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e580b8-c76f-4e68-9f1d-eb619efc0664 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:38.851400 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e580b8-c76f-4e68-9f1d-eb619efc0664 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:38.852871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e580b8-c76f-4e68-9f1d-eb619efc0664 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:38.861859 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 471/953] 10.222.0.15 () {56 vars in 1122 bytes} [Tue Sep 13 01:06:38 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:38.868470 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810d0576-ca34-4a04-98e2-3516a827a29f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:38.935522 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:38.985578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810d0576-ca34-4a04-98e2-3516a827a29f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:39.023965 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef70ef21-7327-4181-9c58-17faa7fd085e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:39.024911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef70ef21-7327-4181-9c58-17faa7fd085e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:39.025389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef70ef21-7327-4181-9c58-17faa7fd085e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:39.025669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef70ef21-7327-4181-9c58-17faa7fd085e None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:39.026299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef70ef21-7327-4181-9c58-17faa7fd085e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:39.026604 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef70ef21-7327-4181-9c58-17faa7fd085e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:39.028010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef70ef21-7327-4181-9c58-17faa7fd085e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:39.033437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 483/954] 10.222.0.15 () {56 vars in 1122 bytes} [Tue Sep 13 01:06:38 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:39.039556 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e580b8-c76f-4e68-9f1d-eb619efc0664 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:39.159310 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e580b8-c76f-4e68-9f1d-eb619efc0664 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:39.198526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e23f86-a3ee-47b6-a63e-5e45826c8c23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:39.199105 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e23f86-a3ee-47b6-a63e-5e45826c8c23 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:39.199388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e23f86-a3ee-47b6-a63e-5e45826c8c23 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:39.199666 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e23f86-a3ee-47b6-a63e-5e45826c8c23 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:39.200433 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e23f86-a3ee-47b6-a63e-5e45826c8c23 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:39.201916 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e23f86-a3ee-47b6-a63e-5e45826c8c23 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:39.214342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:39.235284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 472/955] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:06:39 2022] POST /identity/v3/endpoints => generated 349 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:06:39.309957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebf7dc7-35c2-4e7c-adb3-f3346bde10e5 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:39.310310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebf7dc7-35c2-4e7c-adb3-f3346bde10e5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:39.310507 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebf7dc7-35c2-4e7c-adb3-f3346bde10e5 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:39.312223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 484/956] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:39.318844 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4005bd2f-47b4-4370-bc4a-ad1ad792e46a None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:39.319163 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4005bd2f-47b4-4370-bc4a-ad1ad792e46a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:39.319480 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4005bd2f-47b4-4370-bc4a-ad1ad792e46a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:39.342884 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-4005bd2f-47b4-4370-bc4a-ad1ad792e46a None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:39.344610 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4005bd2f-47b4-4370-bc4a-ad1ad792e46a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:39.436364 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 473/957] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:39 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:39.701074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebf7dc7-35c2-4e7c-adb3-f3346bde10e5 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:39.807812 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebf7dc7-35c2-4e7c-adb3-f3346bde10e5 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:39.830483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:39.849821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cd4eb5-f010-4006-92f8-0416540167b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:39.850713 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cd4eb5-f010-4006-92f8-0416540167b7 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:39.851181 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cd4eb5-f010-4006-92f8-0416540167b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:39.851592 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cd4eb5-f010-4006-92f8-0416540167b7 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:39.852322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1cd4eb5-f010-4006-92f8-0416540167b7 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:39.853879 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1cd4eb5-f010-4006-92f8-0416540167b7 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:39.863600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-e1cd4eb5-f010-4006-92f8-0416540167b7 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 01:06:39.863600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:39.863600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:39.863600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:39.863600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:39.863600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:39.863600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:39.863600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:39.863600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 01:06:39.863600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:39.863600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:39.863600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:39.863600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:39.863600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:39.863600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:39.863600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:39.863600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:39.864945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:39.864945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:39.864945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:39.864945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:39.864945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:39.864945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:39.864945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:39.864945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:39.864945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:39.864945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:39.864945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:39.864945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:39.864945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:39.864945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:39.864945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:39.864945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:39.866039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:39.866039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:39.866039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:39.866039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:39.866039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:39.866039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:39.866039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:39.866039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:39.866039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:39.866039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:39.866039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:39.866039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:39.866039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 01:06:39.866039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:39.868425 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 485/958] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:39 2022] GET /identity/v3/services/compute => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:39.878902 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4005bd2f-47b4-4370-bc4a-ad1ad792e46a None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:39.976898 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4005bd2f-47b4-4370-bc4a-ad1ad792e46a None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:40.000172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:40.012442 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6799faf-4acd-4418-a920-e117c6622fd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:40.013054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6799faf-4acd-4418-a920-e117c6622fd8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:40.013377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6799faf-4acd-4418-a920-e117c6622fd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:40.013645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6799faf-4acd-4418-a920-e117c6622fd8 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:40.014255 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6799faf-4acd-4418-a920-e117c6622fd8 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:40.014600 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6799faf-4acd-4418-a920-e117c6622fd8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:40.016143 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6799faf-4acd-4418-a920-e117c6622fd8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:40.021348 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 474/959] 10.222.0.15 () {56 vars in 1112 bytes} [Tue Sep 13 01:06:39 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:40.026895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cd4eb5-f010-4006-92f8-0416540167b7 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:40.132186 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cd4eb5-f010-4006-92f8-0416540167b7 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:40.168793 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392f3f35-a480-4f3c-b2bf-4d14a47a4e6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:40.169758 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392f3f35-a480-4f3c-b2bf-4d14a47a4e6c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:40.170233 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392f3f35-a480-4f3c-b2bf-4d14a47a4e6c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:40.170698 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392f3f35-a480-4f3c-b2bf-4d14a47a4e6c None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:40.171579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392f3f35-a480-4f3c-b2bf-4d14a47a4e6c None admin] RBAC: Adding query filter params (type=compute) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:40.172074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392f3f35-a480-4f3c-b2bf-4d14a47a4e6c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:40.173635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392f3f35-a480-4f3c-b2bf-4d14a47a4e6c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:40.179331 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 486/960] 10.222.0.15 () {56 vars in 1112 bytes} [Tue Sep 13 01:06:40 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:40.356551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e527d3-846a-4700-8ef5-e4d1a8a77361 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:40.356736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e527d3-846a-4700-8ef5-e4d1a8a77361 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:40.357012 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e527d3-846a-4700-8ef5-e4d1a8a77361 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:40.358486 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 475/961] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:40.363688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae8dbd7-1736-446c-938c-d0b2e7874f0d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:40.363978 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae8dbd7-1736-446c-938c-d0b2e7874f0d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:40.364255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae8dbd7-1736-446c-938c-d0b2e7874f0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:40.383573 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5ae8dbd7-1736-446c-938c-d0b2e7874f0d None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:40.384815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5ae8dbd7-1736-446c-938c-d0b2e7874f0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:40.431711 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:40.462933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 487/962] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:40 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:40.726926 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e527d3-846a-4700-8ef5-e4d1a8a77361 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:40.830344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e527d3-846a-4700-8ef5-e4d1a8a77361 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:40.865452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b18aa31-3578-44fa-953c-b13ad036c088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:40.866086 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b18aa31-3578-44fa-953c-b13ad036c088 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:40.866361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b18aa31-3578-44fa-953c-b13ad036c088 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:40.866629 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b18aa31-3578-44fa-953c-b13ad036c088 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:40.877067 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b18aa31-3578-44fa-953c-b13ad036c088 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:40.882509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-2b18aa31-3578-44fa-953c-b13ad036c088 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 01:06:40.882509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:40.882509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:40.882509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:40.882509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:40.882509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:40.882509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:40.882509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:40.882509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:40.882509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:40.882509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:40.882509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:40.882509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:40.882509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:40.882509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:40.882509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:40.884142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:40.884142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:40.884142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:40.884142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:40.884142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:40.884142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:40.884142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:40.884142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:40.884142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:40.884142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:40.884142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:40.884142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:40.884142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:40.884142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:40.884142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:40.884142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:40.884991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:40.884991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:40.884991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:40.884991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:40.884991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:40.884991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:40.884991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:40.884991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:40.884991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:40.884991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:40.884991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 01:06:40.884991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:40.884991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 476/963] 10.222.0.15 () {56 vars in 1120 bytes} [Tue Sep 13 01:06:40 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:40.891991 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae8dbd7-1736-446c-938c-d0b2e7874f0d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:40.995610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae8dbd7-1736-446c-938c-d0b2e7874f0d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:41.034614 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fb46ac-5acb-4847-81af-eeb54bf5becb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:41.035656 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fb46ac-5acb-4847-81af-eeb54bf5becb None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:41.036158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fb46ac-5acb-4847-81af-eeb54bf5becb None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:41.036717 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fb46ac-5acb-4847-81af-eeb54bf5becb None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:41.037625 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fb46ac-5acb-4847-81af-eeb54bf5becb None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:41.038136 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fb46ac-5acb-4847-81af-eeb54bf5becb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:41.039827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fb46ac-5acb-4847-81af-eeb54bf5becb None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:41.050484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 488/964] 10.222.0.15 () {56 vars in 1115 bytes} [Tue Sep 13 01:06:40 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:41.059930 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b18aa31-3578-44fa-953c-b13ad036c088 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:41.112113 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:41.140008 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeac412-bae6-4583-8f05-f656a418ff45 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:41.140341 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeac412-bae6-4583-8f05-f656a418ff45 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:41.140550 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeac412-bae6-4583-8f05-f656a418ff45 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:41.142152 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 489/965] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:06:41.148909 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5488554d-b989-42ee-affe-1759b4a8b141 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:41.149445 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5488554d-b989-42ee-affe-1759b4a8b141 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:41.149915 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5488554d-b989-42ee-affe-1759b4a8b141 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:41.168297 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5488554d-b989-42ee-affe-1759b4a8b141 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:41.169134 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b18aa31-3578-44fa-953c-b13ad036c088 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:41.169992 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5488554d-b989-42ee-affe-1759b4a8b141 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:41.205497 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28a49aa-35b5-4bbd-88f7-6ea62fe0d6d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:41.206224 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:41.206482 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28a49aa-35b5-4bbd-88f7-6ea62fe0d6d8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:41.207488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28a49aa-35b5-4bbd-88f7-6ea62fe0d6d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:41.207757 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28a49aa-35b5-4bbd-88f7-6ea62fe0d6d8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:41.220196 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28a49aa-35b5-4bbd-88f7-6ea62fe0d6d8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:41.221822 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28a49aa-35b5-4bbd-88f7-6ea62fe0d6d8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:41.237011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-e28a49aa-35b5-4bbd-88f7-6ea62fe0d6d8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:41.237011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:41.237011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:41.237011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:41.237011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:41.237011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:41.237011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:41.237011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:41.237011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:41.237011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:41.237011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:41.237011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:41.237011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:41.237011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:41.237011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:41.237011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:41.237963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:41.237963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:41.237963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:41.237963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:41.237963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:41.237963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:41.237963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:41.237963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:41.237963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:41.237963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:41.237963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:41.237963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:41.237963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:41.237963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:41.237963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:41.237963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:41.237963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:41.238836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:41.238836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:41.238836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:41.241580 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 477/966] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:41.244539 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 490/967] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:41 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 01:06:41.247069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28a49aa-35b5-4bbd-88f7-6ea62fe0d6d8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:41.345662 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:41.351332 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28a49aa-35b5-4bbd-88f7-6ea62fe0d6d8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:41.394055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721db902-4f22-475a-a760-3617eb8ae996 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:41.394999 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721db902-4f22-475a-a760-3617eb8ae996 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:41.395517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721db902-4f22-475a-a760-3617eb8ae996 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:41.396597 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721db902-4f22-475a-a760-3617eb8ae996 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:41.397422 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721db902-4f22-475a-a760-3617eb8ae996 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:41.397895 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721db902-4f22-475a-a760-3617eb8ae996 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:41.399450 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721db902-4f22-475a-a760-3617eb8ae996 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:41.414911 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 478/968] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:41.418473 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5488554d-b989-42ee-affe-1759b4a8b141 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:41.501355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd716990-968b-491d-a817-072d939cb07b None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:41.501962 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd716990-968b-491d-a817-072d939cb07b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:41.502451 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd716990-968b-491d-a817-072d939cb07b None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:41.504274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 479/969] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:41.509679 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa0f930-c029-4aa9-90a5-b4a89010fd96 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:41.510260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa0f930-c029-4aa9-90a5-b4a89010fd96 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:41.510754 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa0f930-c029-4aa9-90a5-b4a89010fd96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:41.529687 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5488554d-b989-42ee-affe-1759b4a8b141 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:41.530486 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-bfa0f930-c029-4aa9-90a5-b4a89010fd96 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:41.532081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-bfa0f930-c029-4aa9-90a5-b4a89010fd96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:41.569310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5099865-af37-4795-88a5-c5dfa0178f72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:41.570795 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5099865-af37-4795-88a5-c5dfa0178f72 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:41.571384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5099865-af37-4795-88a5-c5dfa0178f72 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:41.571901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5099865-af37-4795-88a5-c5dfa0178f72 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:41.579370 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5099865-af37-4795-88a5-c5dfa0178f72 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:41.581151 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5099865-af37-4795-88a5-c5dfa0178f72 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:41.594303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-b5099865-af37-4795-88a5-c5dfa0178f72 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 01:06:41.594303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:41.594303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:41.594303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:41.594303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:41.594303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:41.594303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:41.594303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:41.594303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:41.594303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:41.594303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:41.594303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:41.594303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:41.594303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:41.594303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:41.594303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:41.595353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:41.595353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:41.595353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:41.595353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:41.595353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:41.595353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:41.595353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:41.595353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:41.595353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:41.595353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:41.595353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:41.595353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:41.595353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:41.595353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:41.595353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:41.595353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:41.595353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:41.595353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:41.601772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:41.601772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:41.601772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:41.601772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:41.601772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:41.601772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:41.601772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:41.601772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:41.601772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 01:06:41.601772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:41.601772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 491/970] 10.222.0.15 () {56 vars in 1117 bytes} [Tue Sep 13 01:06:41 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:41.601772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5099865-af37-4795-88a5-c5dfa0178f72 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:41.613537 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 480/971] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:41 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:41.614195 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa0f930-c029-4aa9-90a5-b4a89010fd96 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:41.640883 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:41.716665 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:41.731148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa0f930-c029-4aa9-90a5-b4a89010fd96 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:41.732484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5099865-af37-4795-88a5-c5dfa0178f72 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:41.766467 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc54e6c-075a-4c22-9f1f-68ba67321541 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:41.767441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc54e6c-075a-4c22-9f1f-68ba67321541 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:41.767934 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc54e6c-075a-4c22-9f1f-68ba67321541 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:41.768427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc54e6c-075a-4c22-9f1f-68ba67321541 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:41.769394 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc54e6c-075a-4c22-9f1f-68ba67321541 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:41.769910 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc54e6c-075a-4c22-9f1f-68ba67321541 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:41.772083 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc54e6c-075a-4c22-9f1f-68ba67321541 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:41.776147 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af65ff85-78c8-4430-a07a-e137f3743631 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:41.776743 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af65ff85-78c8-4430-a07a-e137f3743631 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:41.777017 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af65ff85-78c8-4430-a07a-e137f3743631 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:41.777283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af65ff85-78c8-4430-a07a-e137f3743631 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:41.777907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af65ff85-78c8-4430-a07a-e137f3743631 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:41.779326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af65ff85-78c8-4430-a07a-e137f3743631 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:41.782418 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: WARNING keystone.server.flask.common [None req-af65ff85-78c8-4430-a07a-e137f3743631 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 13 01:06:41.793611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 481/972] 10.222.0.15 () {56 vars in 1149 bytes} [Tue Sep 13 01:06:41 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:41.799621 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc54e6c-075a-4c22-9f1f-68ba67321541 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:41.801493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.sql.core [None req-af65ff85-78c8-4430-a07a-e137f3743631 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 13 01:06:41.801493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 13 01:06:41.801493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [parameters: {'user_id': 'd66ab61c202a4752b698fbb9c346cc0d', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 13 01:06:41.801493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81405) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 13 01:06:41.819966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-af65ff85-78c8-4430-a07a-e137f3743631 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 13 01:06:41.819966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:41.819966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 13 01:06:41.819966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 13 01:06:41.819966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 13 01:06:41.819966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 13 01:06:41.819966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 13 01:06:41.819966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application result = self._query(query) Sep 13 01:06:41.819966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 13 01:06:41.819966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application conn.query(q) Sep 13 01:06:41.819966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 13 01:06:41.819966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 13 01:06:41.819966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 13 01:06:41.819966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application result.read() Sep 13 01:06:41.819966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 13 01:06:41.820953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 13 01:06:41.820953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 13 01:06:41.820953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 13 01:06:41.820953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 13 01:06:41.820953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 13 01:06:41.820953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 13 01:06:41.820953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 13 01:06:41.820953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 13 01:06:41.820953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:41.820953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 13 01:06:41.820953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:41.820953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:41.820953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 13 01:06:41.820953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 13 01:06:41.820953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 13 01:06:41.820953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 13 01:06:41.820953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 13 01:06:41.820953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application next(self.gen) Sep 13 01:06:41.821919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 13 01:06:41.821919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application yield resource Sep 13 01:06:41.821919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 13 01:06:41.821919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application next(self.gen) Sep 13 01:06:41.821919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 13 01:06:41.821919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self.session.rollback() Sep 13 01:06:41.821919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 13 01:06:41.821919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self.force_reraise() Sep 13 01:06:41.821919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 13 01:06:41.821919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise self.value Sep 13 01:06:41.821919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 13 01:06:41.821919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 13 01:06:41.821919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 13 01:06:41.821919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application session.commit() Sep 13 01:06:41.821919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 13 01:06:41.821919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 13 01:06:41.821919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 13 01:06:41.822819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._prepare_impl() Sep 13 01:06:41.822819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 13 01:06:41.822819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self.session.flush() Sep 13 01:06:41.822819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 13 01:06:41.822819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._flush(objects) Sep 13 01:06:41.822819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 13 01:06:41.822819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 13 01:06:41.822819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 13 01:06:41.822819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application compat.raise_( Sep 13 01:06:41.822819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 13 01:06:41.822819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception Sep 13 01:06:41.822819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 13 01:06:41.822819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application flush_context.execute() Sep 13 01:06:41.822819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 13 01:06:41.822819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rec.execute(self) Sep 13 01:06:41.822819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 13 01:06:41.822819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 13 01:06:41.823737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 13 01:06:41.823737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 13 01:06:41.823737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 13 01:06:41.823737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 13 01:06:41.823737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 13 01:06:41.823737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 13 01:06:41.823737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 13 01:06:41.823737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 13 01:06:41.823737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 13 01:06:41.823737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 13 01:06:41.823737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 13 01:06:41.823737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 13 01:06:41.823737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 13 01:06:41.823737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 13 01:06:41.823737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 13 01:06:41.823737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception Sep 13 01:06:41.825745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 13 01:06:41.825745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 13 01:06:41.825745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 13 01:06:41.825745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 13 01:06:41.825745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 13 01:06:41.825745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application result = self._query(query) Sep 13 01:06:41.825745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 13 01:06:41.825745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application conn.query(q) Sep 13 01:06:41.825745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 13 01:06:41.825745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 13 01:06:41.825745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 13 01:06:41.825745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application result.read() Sep 13 01:06:41.825745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 13 01:06:41.825745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 13 01:06:41.825745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 13 01:06:41.825745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 13 01:06:41.827608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 13 01:06:41.827608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 13 01:06:41.827608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 13 01:06:41.827608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 13 01:06:41.827608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 13 01:06:41.827608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 13 01:06:41.827608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [parameters: {'user_id': 'd66ab61c202a4752b698fbb9c346cc0d', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 13 01:06:41.827608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 13 01:06:41.827608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:41.827608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:41.827608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:41.827608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:41.827608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:41.827608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:41.827608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:41.829503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:41.829503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:41.829503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:41.829503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:41.829503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:41.829503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:41.829503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:41.829503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 13 01:06:41.829503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 13 01:06:41.829503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:41.829503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:41.829503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:41.829503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:41.829503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:41.829503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:41.829503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 13 01:06:41.829503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 13 01:06:41.831630 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 13 01:06:41.831630 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 13 01:06:41.831630 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 13 01:06:41.831630 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 13 01:06:41.831630 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 13 01:06:41.831630 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:41.839486 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 492/973] 10.222.0.15 () {60 vars in 1131 bytes} [Tue Sep 13 01:06:41 2022] POST /identity/v3/users => generated 160 bytes in 238 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:41.840841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af65ff85-78c8-4430-a07a-e137f3743631 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:41.924661 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc54e6c-075a-4c22-9f1f-68ba67321541 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:41.937155 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:41.948157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af65ff85-78c8-4430-a07a-e137f3743631 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:41.966898 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b5d936-74cd-4cc7-8dd5-beb11e109596 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:41.967516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b5d936-74cd-4cc7-8dd5-beb11e109596 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:41.967789 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b5d936-74cd-4cc7-8dd5-beb11e109596 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:41.968057 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b5d936-74cd-4cc7-8dd5-beb11e109596 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:41.982825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ef1c31-5257-406c-b387-a7e9a44c759c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:41.983490 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ef1c31-5257-406c-b387-a7e9a44c759c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:41.983767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ef1c31-5257-406c-b387-a7e9a44c759c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:41.984036 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ef1c31-5257-406c-b387-a7e9a44c759c None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:41.987617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:41.992953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ef1c31-5257-406c-b387-a7e9a44c759c None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:41.994574 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ef1c31-5257-406c-b387-a7e9a44c759c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:42.005780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-a1ef1c31-5257-406c-b387-a7e9a44c759c None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 01:06:42.005780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:42.005780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:42.005780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:42.005780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:42.005780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:42.005780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:42.005780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:42.005780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:42.005780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:42.005780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:42.005780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:42.005780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:42.005780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:42.005780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:42.005780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:42.006955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:42.006955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:42.006955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:42.006955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:42.006955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:42.006955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:42.006955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:42.006955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:42.006955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:42.006955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:42.006955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:42.006955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:42.006955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:42.006955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:42.006955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:42.006955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:42.006955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:42.006955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:42.007907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:42.007907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:42.007907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:42.007907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:42.007907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:42.007907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:42.007907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:42.007907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:42.007907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 01:06:42.007907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:42.007907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 493/974] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:06:41 2022] GET /identity/v3/users/cinder => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:42.008490 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ef1c31-5257-406c-b387-a7e9a44c759c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:42.054192 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b5d936-74cd-4cc7-8dd5-beb11e109596 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:42.055988 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b5d936-74cd-4cc7-8dd5-beb11e109596 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:42.116221 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ef1c31-5257-406c-b387-a7e9a44c759c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:42.149553 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47410159-e03b-4d9c-9b3e-8d4de99cdd25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:42.150191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47410159-e03b-4d9c-9b3e-8d4de99cdd25 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:42.150471 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47410159-e03b-4d9c-9b3e-8d4de99cdd25 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:42.150744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47410159-e03b-4d9c-9b3e-8d4de99cdd25 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:42.151356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47410159-e03b-4d9c-9b3e-8d4de99cdd25 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:42.152718 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47410159-e03b-4d9c-9b3e-8d4de99cdd25 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:42.162013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-47410159-e03b-4d9c-9b3e-8d4de99cdd25 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 01:06:42.162013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:42.162013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:42.162013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:42.162013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:42.162013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:42.162013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:42.162013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:42.162013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 01:06:42.162013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:42.162013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:42.162013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:42.162013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:42.162013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:42.162013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:42.162013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:42.162013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:42.163139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:42.163139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:42.163139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:42.163139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:42.163139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:42.163139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:42.163139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:42.163139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:42.163139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:42.163139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:42.163139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:42.163139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:42.163139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:42.163139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:42.163139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:42.163139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:42.169259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:42.169259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:42.169259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:42.169259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:42.169259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:42.169259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:42.169259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:42.169259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:42.169259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:42.169259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:42.169259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:42.169259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:42.169259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 01:06:42.169259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:42.170021 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 482/975] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:06:41 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 01:06:42.170021 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b5d936-74cd-4cc7-8dd5-beb11e109596 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:42.170149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 494/976] 10.222.0.15 () {56 vars in 1126 bytes} [Tue Sep 13 01:06:42 2022] GET /identity/v3/services/object-store => generated 93 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:42.170149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47410159-e03b-4d9c-9b3e-8d4de99cdd25 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:42.188088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:42.215082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:42.272203 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b5d936-74cd-4cc7-8dd5-beb11e109596 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:42.274742 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47410159-e03b-4d9c-9b3e-8d4de99cdd25 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:42.310227 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d305a781-d053-428a-99ce-a5f36d8a066f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:42.310760 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d305a781-d053-428a-99ce-a5f36d8a066f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:42.311070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d305a781-d053-428a-99ce-a5f36d8a066f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:42.311343 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d305a781-d053-428a-99ce-a5f36d8a066f None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:42.312046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d305a781-d053-428a-99ce-a5f36d8a066f None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:42.312344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d305a781-d053-428a-99ce-a5f36d8a066f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:42.313492 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f965375e-1911-4578-995c-a32c3d9f078c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:42.314050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d305a781-d053-428a-99ce-a5f36d8a066f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:42.314493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f965375e-1911-4578-995c-a32c3d9f078c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:42.314939 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f965375e-1911-4578-995c-a32c3d9f078c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:42.315351 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f965375e-1911-4578-995c-a32c3d9f078c None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:42.316416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f965375e-1911-4578-995c-a32c3d9f078c None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:42.316721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f965375e-1911-4578-995c-a32c3d9f078c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:42.318656 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f965375e-1911-4578-995c-a32c3d9f078c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:42.332266 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 495/977] 10.222.0.15 () {56 vars in 1122 bytes} [Tue Sep 13 01:06:42 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:42.332266 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f965375e-1911-4578-995c-a32c3d9f078c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:42.362888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 483/978] 10.222.0.15 () {56 vars in 1101 bytes} [Tue Sep 13 01:06:42 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:42.362888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d305a781-d053-428a-99ce-a5f36d8a066f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:42.447515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:42.470574 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f965375e-1911-4578-995c-a32c3d9f078c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:42.482326 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d305a781-d053-428a-99ce-a5f36d8a066f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:42.485988 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:42.513562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65c2c9f-8554-4423-838e-958ce2f214a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:42.514217 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65c2c9f-8554-4423-838e-958ce2f214a8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:42.514589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65c2c9f-8554-4423-838e-958ce2f214a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:42.514763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65c2c9f-8554-4423-838e-958ce2f214a8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:42.525798 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ee792f-4162-43ba-a02b-a4c815a00c1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:42.526577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ee792f-4162-43ba-a02b-a4c815a00c1d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:42.526949 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ee792f-4162-43ba-a02b-a4c815a00c1d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:42.527222 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ee792f-4162-43ba-a02b-a4c815a00c1d None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:42.527322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65c2c9f-8554-4423-838e-958ce2f214a8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:42.527850 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ee792f-4162-43ba-a02b-a4c815a00c1d None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:42.528148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ee792f-4162-43ba-a02b-a4c815a00c1d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:42.528775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65c2c9f-8554-4423-838e-958ce2f214a8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:42.529491 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ee792f-4162-43ba-a02b-a4c815a00c1d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:42.535179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 484/979] 10.222.0.15 () {56 vars in 1122 bytes} [Tue Sep 13 01:06:42 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:42.537148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6931f0cf-d280-40d7-adc0-ff2bb21602a9 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:42.537432 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6931f0cf-d280-40d7-adc0-ff2bb21602a9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:42.537760 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6931f0cf-d280-40d7-adc0-ff2bb21602a9 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:42.539189 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 485/980] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:42.543913 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6931f0cf-d280-40d7-adc0-ff2bb21602a9 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:42.545110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-d65c2c9f-8554-4423-838e-958ce2f214a8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:42.545110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:42.545110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:42.545110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:42.545110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:42.545110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:42.545110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:42.545110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:42.545110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:42.545110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:42.545110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:42.545110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:42.545110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:42.545110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:42.545110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:42.545110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:42.546023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:42.546023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:42.546023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:42.546023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:42.546023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:42.546023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:42.546023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:42.546023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:42.546023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:42.546023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:42.546023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:42.546023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:42.546023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:42.546023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:42.546023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:42.546023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:42.546023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:42.546947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:42.546947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:42.546947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:42.550804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 496/981] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:42.552314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e495d8-bba1-40a0-a7d1-2b92e324d607 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:42.552593 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e495d8-bba1-40a0-a7d1-2b92e324d607 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:42.553551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e495d8-bba1-40a0-a7d1-2b92e324d607 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:42.579619 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-09e495d8-bba1-40a0-a7d1-2b92e324d607 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:42.581034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-09e495d8-bba1-40a0-a7d1-2b92e324d607 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:42.656506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6931f0cf-d280-40d7-adc0-ff2bb21602a9 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:42.657256 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 497/982] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:42 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:42.658290 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e495d8-bba1-40a0-a7d1-2b92e324d607 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:42.694562 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17968825-44c9-4de6-a201-f7237e7af8ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:42.695362 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17968825-44c9-4de6-a201-f7237e7af8ec None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:42.695649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17968825-44c9-4de6-a201-f7237e7af8ec None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:42.696645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17968825-44c9-4de6-a201-f7237e7af8ec None admin] PATH_INFO: `/v3/endpoints` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:42.697371 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17968825-44c9-4de6-a201-f7237e7af8ec None admin] RBAC: Adding query filter params (interface=admin, service_id=12f35fe97fcf4d6a828249b14c3e6aae, region_id=RegionOne) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:42.697687 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17968825-44c9-4de6-a201-f7237e7af8ec None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:42.699130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17968825-44c9-4de6-a201-f7237e7af8ec None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:42.705528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 486/983] 10.222.0.15 () {56 vars in 1249 bytes} [Tue Sep 13 01:06:42 2022] GET /identity/v3/endpoints?service_id=12f35fe97fcf4d6a828249b14c3e6aae&interface=admin®ion_id=RegionOne => generated 193 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:42.711425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17968825-44c9-4de6-a201-f7237e7af8ec None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:42.724483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:42.744966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:42.765614 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e495d8-bba1-40a0-a7d1-2b92e324d607 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:42.803477 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1564f5f-4411-4c0c-bbf1-069061bd464d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:42.804091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1564f5f-4411-4c0c-bbf1-069061bd464d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:42.804368 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1564f5f-4411-4c0c-bbf1-069061bd464d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:42.804657 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1564f5f-4411-4c0c-bbf1-069061bd464d None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:42.805267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1564f5f-4411-4c0c-bbf1-069061bd464d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:42.805568 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1564f5f-4411-4c0c-bbf1-069061bd464d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:42.806929 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1564f5f-4411-4c0c-bbf1-069061bd464d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:42.818897 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17968825-44c9-4de6-a201-f7237e7af8ec None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:42.819134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 498/984] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:42.827072 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1564f5f-4411-4c0c-bbf1-069061bd464d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:42.862979 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20def648-2ec5-4f1c-9ffc-4a35e1787934 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:42.863896 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20def648-2ec5-4f1c-9ffc-4a35e1787934 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:42.864383 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20def648-2ec5-4f1c-9ffc-4a35e1787934 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:42.865394 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20def648-2ec5-4f1c-9ffc-4a35e1787934 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:42.866177 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20def648-2ec5-4f1c-9ffc-4a35e1787934 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:42.867750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20def648-2ec5-4f1c-9ffc-4a35e1787934 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:42.882894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 487/985] 10.222.0.15 () {56 vars in 1087 bytes} [Tue Sep 13 01:06:42 2022] GET /identity/v3/services => generated 1497 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:06:42.934184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20def648-2ec5-4f1c-9ffc-4a35e1787934 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:42.951678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1564f5f-4411-4c0c-bbf1-069061bd464d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:42.976984 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:42.990660 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2676c3-e318-40da-9599-1d63ee9367c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:42.991355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2676c3-e318-40da-9599-1d63ee9367c0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:42.991632 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2676c3-e318-40da-9599-1d63ee9367c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:42.991903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2676c3-e318-40da-9599-1d63ee9367c0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:42.999143 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2676c3-e318-40da-9599-1d63ee9367c0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:43.000619 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2676c3-e318-40da-9599-1d63ee9367c0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:43.013371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-ba2676c3-e318-40da-9599-1d63ee9367c0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:43.013371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:43.013371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:43.013371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:43.013371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:43.013371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:43.013371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:43.013371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:43.013371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:43.013371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:43.013371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:43.013371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:43.013371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:43.013371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:43.013371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:43.014532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:43.014532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:43.014532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:43.014532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:43.014532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:43.014532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:43.014532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:43.014532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:43.014532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:43.014532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:43.014532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:43.014532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:43.014532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:43.014532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:43.014532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:43.014532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:43.014532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:43.015582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:43.015582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:43.015582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:43.015582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:43.015582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:43.015582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:43.015582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:43.015582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:43.015582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 499/986] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:43.020520 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2676c3-e318-40da-9599-1d63ee9367c0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:43.056565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:43.065634 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20def648-2ec5-4f1c-9ffc-4a35e1787934 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:43.104277 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e55ba9-6040-4afc-97f2-a1202472b2a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:43.104888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e55ba9-6040-4afc-97f2-a1202472b2a1 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:43.105168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e55ba9-6040-4afc-97f2-a1202472b2a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:43.105440 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e55ba9-6040-4afc-97f2-a1202472b2a1 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:43.106085 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e55ba9-6040-4afc-97f2-a1202472b2a1 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:43.107472 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e55ba9-6040-4afc-97f2-a1202472b2a1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:43.123144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 488/987] 10.222.0.15 () {60 vars in 1141 bytes} [Tue Sep 13 01:06:42 2022] POST /identity/v3/services => generated 242 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:06:43.134602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2676c3-e318-40da-9599-1d63ee9367c0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:43.181092 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3e0ae4-ad66-4448-9517-254a7f8b2160 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:43.181726 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3e0ae4-ad66-4448-9517-254a7f8b2160 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:43.182000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3e0ae4-ad66-4448-9517-254a7f8b2160 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:43.182267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3e0ae4-ad66-4448-9517-254a7f8b2160 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:43.182996 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3e0ae4-ad66-4448-9517-254a7f8b2160 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:43.183301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3e0ae4-ad66-4448-9517-254a7f8b2160 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:43.184809 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3e0ae4-ad66-4448-9517-254a7f8b2160 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:43.202887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 500/988] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:43.205069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e55ba9-6040-4afc-97f2-a1202472b2a1 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:43.317350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e55ba9-6040-4afc-97f2-a1202472b2a1 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:43.353537 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd2d34f-4887-44d5-a9b5-e398ac9eb7e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:43.354154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd2d34f-4887-44d5-a9b5-e398ac9eb7e1 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:43.354431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd2d34f-4887-44d5-a9b5-e398ac9eb7e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:43.354702 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd2d34f-4887-44d5-a9b5-e398ac9eb7e1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:43.355509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd2d34f-4887-44d5-a9b5-e398ac9eb7e1 None admin] RBAC: Adding query filter params (role.id=11f2e919cae541428f8c564b7b67eea0, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=e28a0ba45389417ba8ae92376dba87ff) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:43.355820 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd2d34f-4887-44d5-a9b5-e398ac9eb7e1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:43.357246 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd2d34f-4887-44d5-a9b5-e398ac9eb7e1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:43.363907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 489/989] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:06:43 2022] GET /identity/v3/role_assignments?user.id=e28a0ba45389417ba8ae92376dba87ff&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=11f2e919cae541428f8c564b7b67eea0 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:45.047413 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abecfbe-2645-443f-84e6-3a699dc12e96 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:45.048080 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abecfbe-2645-443f-84e6-3a699dc12e96 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:45.048080 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abecfbe-2645-443f-84e6-3a699dc12e96 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:45.049444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 501/990] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:45.055173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64c13fe-3bc8-45f8-a30c-97686b5b4988 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:45.055581 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64c13fe-3bc8-45f8-a30c-97686b5b4988 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:45.055748 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64c13fe-3bc8-45f8-a30c-97686b5b4988 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:45.073957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e64c13fe-3bc8-45f8-a30c-97686b5b4988 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:45.075193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e64c13fe-3bc8-45f8-a30c-97686b5b4988 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:45.089943 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:45.146130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 490/991] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:45 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:45.291535 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e622a1-075c-4eba-9395-e0b0b17eaac6 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:45.291875 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e622a1-075c-4eba-9395-e0b0b17eaac6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:45.292090 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e622a1-075c-4eba-9395-e0b0b17eaac6 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:45.293557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 502/992] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:45.298634 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31d2461-0068-4d07-87af-a2253c0a39b0 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:45.298894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31d2461-0068-4d07-87af-a2253c0a39b0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:45.299179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31d2461-0068-4d07-87af-a2253c0a39b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:45.318253 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f31d2461-0068-4d07-87af-a2253c0a39b0 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:45.319456 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f31d2461-0068-4d07-87af-a2253c0a39b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:45.385162 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 491/993] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:45 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:45.417660 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e622a1-075c-4eba-9395-e0b0b17eaac6 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:45.461887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:45.527359 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e622a1-075c-4eba-9395-e0b0b17eaac6 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:45.560996 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8246ff19-0c7d-42cf-85d7-7fd2eb3e4837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:45.561551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8246ff19-0c7d-42cf-85d7-7fd2eb3e4837 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:45.561829 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8246ff19-0c7d-42cf-85d7-7fd2eb3e4837 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:45.562098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8246ff19-0c7d-42cf-85d7-7fd2eb3e4837 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:45.562940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8246ff19-0c7d-42cf-85d7-7fd2eb3e4837 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:45.564438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8246ff19-0c7d-42cf-85d7-7fd2eb3e4837 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:45.573078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-8246ff19-0c7d-42cf-85d7-7fd2eb3e4837 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 01:06:45.573078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:45.573078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:45.573078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:45.573078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:45.573078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:45.573078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:45.573078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:45.573078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 01:06:45.573078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:45.573078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:45.573078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:45.573078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:45.573078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:45.573078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:45.573078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:45.573078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:45.574200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:45.574200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:45.574200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:45.574200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:45.574200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:45.574200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:45.574200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:45.574200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:45.574200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:45.574200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:45.574200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:45.574200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:45.574200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:45.574200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:45.574200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:45.574200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:45.575139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:45.575139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:45.575139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:45.575139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:45.575139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:45.575139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:45.575139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:45.575139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:45.575139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:45.575139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:45.575139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:45.575139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:45.575139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 01:06:45.575139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:45.579049 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 503/994] 10.222.0.15 () {56 vars in 1126 bytes} [Tue Sep 13 01:06:45 2022] GET /identity/v3/services/object-store => generated 93 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:45.580294 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31d2461-0068-4d07-87af-a2253c0a39b0 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:45.584972 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6705c5a0-b541-4d4a-892d-3ac71d0ea823 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:45.585256 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6705c5a0-b541-4d4a-892d-3ac71d0ea823 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:45.585528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6705c5a0-b541-4d4a-892d-3ac71d0ea823 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:45.586986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 504/995] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:06:45.592785 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00af6968-5f65-4890-9a8c-2a275b0adf76 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:45.593234 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00af6968-5f65-4890-9a8c-2a275b0adf76 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:45.593698 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00af6968-5f65-4890-9a8c-2a275b0adf76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:45.615572 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-00af6968-5f65-4890-9a8c-2a275b0adf76 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:45.617090 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-00af6968-5f65-4890-9a8c-2a275b0adf76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:45.663464 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:45.682804 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31d2461-0068-4d07-87af-a2253c0a39b0 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:45.686831 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 505/996] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:45 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 01:06:45.691392 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00af6968-5f65-4890-9a8c-2a275b0adf76 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:45.718238 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:45.722109 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c2af0f-fca6-4a71-9c1b-951b5fc7bd38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:45.723495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c2af0f-fca6-4a71-9c1b-951b5fc7bd38 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:45.724395 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c2af0f-fca6-4a71-9c1b-951b5fc7bd38 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:45.724669 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c2af0f-fca6-4a71-9c1b-951b5fc7bd38 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:45.725297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c2af0f-fca6-4a71-9c1b-951b5fc7bd38 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:45.725596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c2af0f-fca6-4a71-9c1b-951b5fc7bd38 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:45.727004 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c2af0f-fca6-4a71-9c1b-951b5fc7bd38 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:45.735016 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 492/997] 10.222.0.15 () {56 vars in 1122 bytes} [Tue Sep 13 01:06:45 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:45.738717 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c2af0f-fca6-4a71-9c1b-951b5fc7bd38 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:45.789191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00af6968-5f65-4890-9a8c-2a275b0adf76 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:45.824790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2974a510-5c76-4404-8d59-3c57d69be4da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:45.825342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2974a510-5c76-4404-8d59-3c57d69be4da None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:45.825620 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2974a510-5c76-4404-8d59-3c57d69be4da None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:45.825890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2974a510-5c76-4404-8d59-3c57d69be4da None admin] PATH_INFO: `/v3/services/compute` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:45.826460 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2974a510-5c76-4404-8d59-3c57d69be4da None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:45.827842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2974a510-5c76-4404-8d59-3c57d69be4da None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:45.836597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-2974a510-5c76-4404-8d59-3c57d69be4da None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 01:06:45.836597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:45.836597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:45.836597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:45.836597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:45.836597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:45.836597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:45.836597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:45.836597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 01:06:45.836597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:45.836597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:45.836597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:45.836597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:45.836597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:45.836597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:45.836597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:45.836597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:45.837589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:45.837589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:45.837589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:45.837589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:45.837589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:45.837589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:45.837589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:45.837589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:45.837589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:45.837589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:45.837589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:45.837589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:45.837589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:45.837589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:45.837589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:45.837589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:45.838411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:45.838411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:45.838411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:45.838411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:45.838411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:45.838411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:45.838411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:45.838411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:45.838411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:45.838411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:45.838411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:45.838411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:45.838411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 01:06:45.838411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:45.838411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 506/998] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:45 2022] GET /identity/v3/services/compute => generated 88 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:45.841856 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c2af0f-fca6-4a71-9c1b-951b5fc7bd38 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:45.843350 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2974a510-5c76-4404-8d59-3c57d69be4da None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:45.877341 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6bc07d-3819-4244-b0b9-49e5dfa0c250 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:45.877941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6bc07d-3819-4244-b0b9-49e5dfa0c250 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:45.878224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6bc07d-3819-4244-b0b9-49e5dfa0c250 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:45.878489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6bc07d-3819-4244-b0b9-49e5dfa0c250 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:45.879125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6bc07d-3819-4244-b0b9-49e5dfa0c250 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:45.879435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6bc07d-3819-4244-b0b9-49e5dfa0c250 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:45.881463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6bc07d-3819-4244-b0b9-49e5dfa0c250 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:45.886372 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 493/999] 10.222.0.15 () {56 vars in 1122 bytes} [Tue Sep 13 01:06:45 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:45.891687 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6bc07d-3819-4244-b0b9-49e5dfa0c250 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:45.902276 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:45.941678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:45.949007 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2974a510-5c76-4404-8d59-3c57d69be4da None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:45.983393 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062d1132-46fb-4c90-b6e9-d6f6ca0d0114 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:45.984000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062d1132-46fb-4c90-b6e9-d6f6ca0d0114 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:45.984275 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062d1132-46fb-4c90-b6e9-d6f6ca0d0114 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:45.984555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062d1132-46fb-4c90-b6e9-d6f6ca0d0114 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:45.985175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062d1132-46fb-4c90-b6e9-d6f6ca0d0114 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:45.985475 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062d1132-46fb-4c90-b6e9-d6f6ca0d0114 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:45.986826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062d1132-46fb-4c90-b6e9-d6f6ca0d0114 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:45.992123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 507/1000] 10.222.0.15 () {56 vars in 1112 bytes} [Tue Sep 13 01:06:45 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:45.997472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062d1132-46fb-4c90-b6e9-d6f6ca0d0114 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:45.999495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6bc07d-3819-4244-b0b9-49e5dfa0c250 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:46.033148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9a0b9c-ca1e-4c97-9c64-48da498d212f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:46.033748 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9a0b9c-ca1e-4c97-9c64-48da498d212f None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:46.034028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9a0b9c-ca1e-4c97-9c64-48da498d212f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:46.034299 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9a0b9c-ca1e-4c97-9c64-48da498d212f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:46.034980 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9a0b9c-ca1e-4c97-9c64-48da498d212f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:46.036833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9a0b9c-ca1e-4c97-9c64-48da498d212f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:46.054414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 494/1001] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:06:45 2022] POST /identity/v3/endpoints => generated 325 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:06:46.055249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9a0b9c-ca1e-4c97-9c64-48da498d212f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:46.116975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062d1132-46fb-4c90-b6e9-d6f6ca0d0114 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:46.130468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:46.165790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37cbc8b9-b01c-4265-8648-b311d65d8d09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:46.166490 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cbc8b9-b01c-4265-8648-b311d65d8d09 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:46.166772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cbc8b9-b01c-4265-8648-b311d65d8d09 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:46.167095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cbc8b9-b01c-4265-8648-b311d65d8d09 None admin] PATH_INFO: `/v3/services/image` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:46.167708 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37cbc8b9-b01c-4265-8648-b311d65d8d09 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:46.169183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37cbc8b9-b01c-4265-8648-b311d65d8d09 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:46.170422 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9a0b9c-ca1e-4c97-9c64-48da498d212f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:46.179825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-37cbc8b9-b01c-4265-8648-b311d65d8d09 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 01:06:46.179825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:46.179825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:46.179825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:46.179825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:46.179825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:46.179825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:46.179825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:46.179825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 01:06:46.179825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:46.179825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:46.179825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:46.179825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:46.179825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:46.179825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:46.179825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:46.179825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:46.180947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:46.180947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:46.180947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:46.180947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:46.180947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:46.180947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:46.180947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:46.180947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:46.180947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:46.180947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:46.180947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:46.180947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:46.180947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:46.180947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:46.180947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:46.180947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:46.181876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:46.181876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:46.181876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:46.181876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:46.181876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:46.181876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:46.181876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:46.181876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:46.181876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:46.181876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:46.181876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:46.181876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:46.181876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 01:06:46.181876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:46.181876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 508/1002] 10.222.0.15 () {56 vars in 1105 bytes} [Tue Sep 13 01:06:45 2022] GET /identity/v3/services/image => generated 86 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:46.187342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37cbc8b9-b01c-4265-8648-b311d65d8d09 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:46.211390 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8780c0b3-55c3-4180-bc38-9bc3210f12db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:46.212027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8780c0b3-55c3-4180-bc38-9bc3210f12db None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:46.212220 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:46.212324 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8780c0b3-55c3-4180-bc38-9bc3210f12db None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:46.212558 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8780c0b3-55c3-4180-bc38-9bc3210f12db None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:46.213351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8780c0b3-55c3-4180-bc38-9bc3210f12db None admin] RBAC: Adding query filter params (type=compute) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:46.213559 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8780c0b3-55c3-4180-bc38-9bc3210f12db None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:46.215093 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8780c0b3-55c3-4180-bc38-9bc3210f12db None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:46.220476 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 495/1003] 10.222.0.15 () {56 vars in 1112 bytes} [Tue Sep 13 01:06:46 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:46.226377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8780c0b3-55c3-4180-bc38-9bc3210f12db None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:46.306094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37cbc8b9-b01c-4265-8648-b311d65d8d09 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:46.333635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8780c0b3-55c3-4180-bc38-9bc3210f12db None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:46.346927 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7581a24e-187c-478e-892b-224f2e3070d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:46.348045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7581a24e-187c-478e-892b-224f2e3070d6 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:46.348747 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7581a24e-187c-478e-892b-224f2e3070d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:46.349119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7581a24e-187c-478e-892b-224f2e3070d6 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:46.349846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7581a24e-187c-478e-892b-224f2e3070d6 None admin] RBAC: Adding query filter params (name=image) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:46.350215 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7581a24e-187c-478e-892b-224f2e3070d6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:46.351680 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7581a24e-187c-478e-892b-224f2e3070d6 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:46.361404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 509/1004] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:46 2022] GET /identity/v3/services?name=image => generated 122 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:46.362599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7581a24e-187c-478e-892b-224f2e3070d6 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:46.363072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:46.369615 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1a35c7-6419-42e7-bef0-af0de3d27c16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:46.370236 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1a35c7-6419-42e7-bef0-af0de3d27c16 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:46.370582 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1a35c7-6419-42e7-bef0-af0de3d27c16 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:46.370959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1a35c7-6419-42e7-bef0-af0de3d27c16 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:46.371717 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1a35c7-6419-42e7-bef0-af0de3d27c16 None admin] RBAC: Adding query filter params (interface=public, service_id=74cfd629f8fc4d70a7ffc1cb4f158341, region_id=RegionOne) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:46.372093 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1a35c7-6419-42e7-bef0-af0de3d27c16 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:46.373509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1a35c7-6419-42e7-bef0-af0de3d27c16 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:46.383202 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 496/1005] 10.222.0.15 () {56 vars in 1251 bytes} [Tue Sep 13 01:06:46 2022] GET /identity/v3/endpoints?service_id=74cfd629f8fc4d70a7ffc1cb4f158341&interface=public®ion_id=RegionOne => generated 194 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:46.384251 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1a35c7-6419-42e7-bef0-af0de3d27c16 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:46.462435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:46.478958 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7581a24e-187c-478e-892b-224f2e3070d6 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:46.500069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1a35c7-6419-42e7-bef0-af0de3d27c16 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:46.513413 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8632eab-409d-4fce-9a54-7438c2d6c68a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:46.514023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8632eab-409d-4fce-9a54-7438c2d6c68a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:46.514296 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8632eab-409d-4fce-9a54-7438c2d6c68a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:46.514565 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8632eab-409d-4fce-9a54-7438c2d6c68a None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:46.515217 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8632eab-409d-4fce-9a54-7438c2d6c68a None admin] RBAC: Adding query filter params (type=image) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:46.515516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8632eab-409d-4fce-9a54-7438c2d6c68a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:46.516866 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8632eab-409d-4fce-9a54-7438c2d6c68a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:46.521882 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 510/1006] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:46 2022] GET /identity/v3/services?type=image => generated 122 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:46.548424 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9f7bf6-3898-445a-a9e6-778eb62b010d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:46.549170 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9f7bf6-3898-445a-a9e6-778eb62b010d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:46.549448 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9f7bf6-3898-445a-a9e6-778eb62b010d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:46.549719 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9f7bf6-3898-445a-a9e6-778eb62b010d None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:46.550329 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9f7bf6-3898-445a-a9e6-778eb62b010d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:46.552419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9f7bf6-3898-445a-a9e6-778eb62b010d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:46.563109 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 497/1007] 10.222.0.15 () {56 vars in 1087 bytes} [Tue Sep 13 01:06:46 2022] GET /identity/v3/services => generated 1727 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:06:48.225155 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c74da28-e5b5-4bad-a02f-5a1b4d66877b None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:48.225685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c74da28-e5b5-4bad-a02f-5a1b4d66877b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:48.225685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c74da28-e5b5-4bad-a02f-5a1b4d66877b None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:48.227186 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 511/1008] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:06:48.232304 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fe66b9-0b59-43b0-b35b-1aa3552e3cb7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:48.232702 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fe66b9-0b59-43b0-b35b-1aa3552e3cb7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:48.232861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fe66b9-0b59-43b0-b35b-1aa3552e3cb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:48.251170 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-49fe66b9-0b59-43b0-b35b-1aa3552e3cb7 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:48.252354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-49fe66b9-0b59-43b0-b35b-1aa3552e3cb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:48.309297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:48.322942 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 498/1009] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:48 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:48.592163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c74da28-e5b5-4bad-a02f-5a1b4d66877b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:48.697544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d891c28f-9fa6-42eb-88c6-79a56a47035c None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:48.697941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d891c28f-9fa6-42eb-88c6-79a56a47035c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:48.698231 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d891c28f-9fa6-42eb-88c6-79a56a47035c None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:48.699604 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c74da28-e5b5-4bad-a02f-5a1b4d66877b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:48.699720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 499/1010] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:48.704466 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdb794d-c781-4305-b8fa-f9fa309e0ee3 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:48.704751 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdb794d-c781-4305-b8fa-f9fa309e0ee3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:48.705028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdb794d-c781-4305-b8fa-f9fa309e0ee3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:48.726067 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ccdb794d-c781-4305-b8fa-f9fa309e0ee3 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:48.727293 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ccdb794d-c781-4305-b8fa-f9fa309e0ee3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:48.733353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e59bc2-93d2-42a5-9822-4e9f3fc800d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:48.733982 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e59bc2-93d2-42a5-9822-4e9f3fc800d0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:48.734254 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e59bc2-93d2-42a5-9822-4e9f3fc800d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:48.734519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e59bc2-93d2-42a5-9822-4e9f3fc800d0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:48.735625 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e59bc2-93d2-42a5-9822-4e9f3fc800d0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:48.737016 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e59bc2-93d2-42a5-9822-4e9f3fc800d0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:48.738560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 512/1011] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:48.740350 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6824b322-9c1b-40f7-b21e-a94a3279b79c None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:48.740640 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6824b322-9c1b-40f7-b21e-a94a3279b79c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:48.740912 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6824b322-9c1b-40f7-b21e-a94a3279b79c None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:48.742213 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 513/1012] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:48.744772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6824b322-9c1b-40f7-b21e-a94a3279b79c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:48.763042 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:48.796982 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 500/1013] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:48 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:48.798216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc555749-14d4-475b-84e1-89fb0b31d57f None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:48.798506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc555749-14d4-475b-84e1-89fb0b31d57f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:48.798783 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc555749-14d4-475b-84e1-89fb0b31d57f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:48.818365 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-dc555749-14d4-475b-84e1-89fb0b31d57f None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:48.819607 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-dc555749-14d4-475b-84e1-89fb0b31d57f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:48.848082 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6824b322-9c1b-40f7-b21e-a94a3279b79c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:48.884309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1d91bc-a646-44d1-b0e5-706b4e0ad5c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:48.884917 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1d91bc-a646-44d1-b0e5-706b4e0ad5c5 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:48.885198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1d91bc-a646-44d1-b0e5-706b4e0ad5c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:48.885468 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1d91bc-a646-44d1-b0e5-706b4e0ad5c5 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:48.886111 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1d91bc-a646-44d1-b0e5-706b4e0ad5c5 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:48.887559 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1d91bc-a646-44d1-b0e5-706b4e0ad5c5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:48.890181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 501/1014] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:48 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:48.922108 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 514/1015] 10.222.0.15 () {60 vars in 1141 bytes} [Tue Sep 13 01:06:48 2022] POST /identity/v3/projects => generated 311 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:06:49.072196 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc555749-14d4-475b-84e1-89fb0b31d57f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:49.130227 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:49.163617 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1d91bc-a646-44d1-b0e5-706b4e0ad5c5 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:49.182312 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc555749-14d4-475b-84e1-89fb0b31d57f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:49.220574 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bf7d76-7f07-4d49-a31a-f89124c85d36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:49.221534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bf7d76-7f07-4d49-a31a-f89124c85d36 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:49.222480 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bf7d76-7f07-4d49-a31a-f89124c85d36 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:49.222973 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bf7d76-7f07-4d49-a31a-f89124c85d36 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:49.223706 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bf7d76-7f07-4d49-a31a-f89124c85d36 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:49.225077 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bf7d76-7f07-4d49-a31a-f89124c85d36 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:49.240980 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 502/1016] 10.222.0.15 () {60 vars in 1141 bytes} [Tue Sep 13 01:06:49 2022] POST /identity/v3/services => generated 242 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:06:49.246937 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:49.289541 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1d91bc-a646-44d1-b0e5-706b4e0ad5c5 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:49.334908 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc23cdb-9e30-456e-98aa-91380bf4bada None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:49.335626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc23cdb-9e30-456e-98aa-91380bf4bada None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:49.335910 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc23cdb-9e30-456e-98aa-91380bf4bada None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:49.336177 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc23cdb-9e30-456e-98aa-91380bf4bada None admin] PATH_INFO: `/v3/services/compute` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:49.336826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc23cdb-9e30-456e-98aa-91380bf4bada None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:49.338270 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc23cdb-9e30-456e-98aa-91380bf4bada None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:49.348196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-3cc23cdb-9e30-456e-98aa-91380bf4bada None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 01:06:49.348196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:49.348196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:49.348196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:49.348196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:49.348196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:49.348196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:49.348196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:49.348196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 01:06:49.348196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:49.348196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:49.348196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:49.348196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:49.348196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:49.348196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:49.348196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:49.348196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:49.349851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:49.349851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:49.349851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:49.349851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:49.349851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:49.349851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:49.349851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:49.349851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:49.349851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:49.349851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:49.349851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:49.349851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:49.349851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:49.349851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:49.349851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:49.349851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:49.350715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:49.350715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:49.350715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:49.350715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:49.350715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:49.350715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:49.350715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:49.350715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:49.350715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:49.350715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:49.350715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:49.350715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:49.350715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 01:06:49.350715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:49.355138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 515/1017] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:49 2022] GET /identity/v3/services/compute => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:49.356471 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bf7d76-7f07-4d49-a31a-f89124c85d36 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:49.476409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bf7d76-7f07-4d49-a31a-f89124c85d36 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:49.487665 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:49.513555 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c5e206-bd9d-461a-a166-7019fb27bdf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:49.514170 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c5e206-bd9d-461a-a166-7019fb27bdf8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:49.514452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c5e206-bd9d-461a-a166-7019fb27bdf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:49.514727 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c5e206-bd9d-461a-a166-7019fb27bdf8 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:49.515385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c5e206-bd9d-461a-a166-7019fb27bdf8 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:49.515691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c5e206-bd9d-461a-a166-7019fb27bdf8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:49.517148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c5e206-bd9d-461a-a166-7019fb27bdf8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:49.527326 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 503/1018] 10.222.0.15 () {56 vars in 1112 bytes} [Tue Sep 13 01:06:49 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:49.528868 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc23cdb-9e30-456e-98aa-91380bf4bada None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:49.631478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc23cdb-9e30-456e-98aa-91380bf4bada None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:49.657860 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:49.667355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d236ba62-a90b-4f31-b75d-93a3e667619e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:49.667899 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d236ba62-a90b-4f31-b75d-93a3e667619e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:49.668173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d236ba62-a90b-4f31-b75d-93a3e667619e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:49.668439 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d236ba62-a90b-4f31-b75d-93a3e667619e None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:49.669053 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d236ba62-a90b-4f31-b75d-93a3e667619e None admin] RBAC: Adding query filter params (type=compute) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:49.669347 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d236ba62-a90b-4f31-b75d-93a3e667619e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:49.670686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d236ba62-a90b-4f31-b75d-93a3e667619e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:49.680876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 516/1019] 10.222.0.15 () {56 vars in 1112 bytes} [Tue Sep 13 01:06:49 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:49.682193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c5e206-bd9d-461a-a166-7019fb27bdf8 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:49.787259 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c5e206-bd9d-461a-a166-7019fb27bdf8 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:49.822092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968576fe-eb0a-401e-a95f-b680966385f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:49.823056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968576fe-eb0a-401e-a95f-b680966385f1 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:49.823556 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968576fe-eb0a-401e-a95f-b680966385f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:49.824019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968576fe-eb0a-401e-a95f-b680966385f1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:49.824951 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968576fe-eb0a-401e-a95f-b680966385f1 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:49.826493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968576fe-eb0a-401e-a95f-b680966385f1 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:49.847468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 504/1020] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:06:49 2022] POST /identity/v3/endpoints => generated 334 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:06:51.136979 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7638bc85-d317-44ca-af62-e35386711bc6 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:51.137484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7638bc85-d317-44ca-af62-e35386711bc6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:51.137560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7638bc85-d317-44ca-af62-e35386711bc6 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:51.139024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 517/1021] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:51.144297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862cf3a5-332e-4477-9efc-1dca3bd4edf5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:51.144702 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862cf3a5-332e-4477-9efc-1dca3bd4edf5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:51.144870 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862cf3a5-332e-4477-9efc-1dca3bd4edf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:51.164209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-862cf3a5-332e-4477-9efc-1dca3bd4edf5 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:51.165560 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-862cf3a5-332e-4477-9efc-1dca3bd4edf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:51.190469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:51.242956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 505/1022] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:51 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:51.409733 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db14eee-b3c8-4ab9-ad1e-9c1f5a4ca569 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:51.410030 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db14eee-b3c8-4ab9-ad1e-9c1f5a4ca569 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:51.410224 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db14eee-b3c8-4ab9-ad1e-9c1f5a4ca569 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:51.412371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 518/1023] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:51.417508 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e801b5-43da-434d-929e-8d2146a590cb None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:51.417820 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e801b5-43da-434d-929e-8d2146a590cb None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:51.418098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e801b5-43da-434d-929e-8d2146a590cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:51.437058 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c8e801b5-43da-434d-929e-8d2146a590cb None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:51.438491 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c8e801b5-43da-434d-929e-8d2146a590cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:51.508125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 506/1024] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:51 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:51.510144 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db14eee-b3c8-4ab9-ad1e-9c1f5a4ca569 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:51.610382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db14eee-b3c8-4ab9-ad1e-9c1f5a4ca569 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:51.643596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5690011-a47e-4009-905b-074ea2735b8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:51.644218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5690011-a47e-4009-905b-074ea2735b8c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:51.644493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5690011-a47e-4009-905b-074ea2735b8c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:51.644759 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5690011-a47e-4009-905b-074ea2735b8c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:51.646371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5690011-a47e-4009-905b-074ea2735b8c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:51.647857 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5690011-a47e-4009-905b-074ea2735b8c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:51.649553 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 519/1025] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:51.655161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e801b5-43da-434d-929e-8d2146a590cb None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:51.738331 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:51.754456 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e801b5-43da-434d-929e-8d2146a590cb None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:51.776122 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5690011-a47e-4009-905b-074ea2735b8c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:51.794301 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831029e0-9570-476f-a2d9-98dea24ff8dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:51.794941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831029e0-9570-476f-a2d9-98dea24ff8dd None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:51.795223 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831029e0-9570-476f-a2d9-98dea24ff8dd None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:51.795496 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831029e0-9570-476f-a2d9-98dea24ff8dd None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:51.796145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831029e0-9570-476f-a2d9-98dea24ff8dd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:51.797653 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831029e0-9570-476f-a2d9-98dea24ff8dd None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:51.822906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 507/1026] 10.222.0.15 () {60 vars in 1132 bytes} [Tue Sep 13 01:06:51 2022] POST /identity/v3/users => generated 285 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:06:51.843146 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:51.889681 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5690011-a47e-4009-905b-074ea2735b8c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:51.927424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c77d696-d1fb-43c1-b496-34974375e5da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:51.928077 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c77d696-d1fb-43c1-b496-34974375e5da None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:51.928354 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c77d696-d1fb-43c1-b496-34974375e5da None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:51.928622 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c77d696-d1fb-43c1-b496-34974375e5da None admin] PATH_INFO: `/v3/services/image` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:51.929201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c77d696-d1fb-43c1-b496-34974375e5da None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:51.930577 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c77d696-d1fb-43c1-b496-34974375e5da None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:51.940293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-4c77d696-d1fb-43c1-b496-34974375e5da None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 01:06:51.940293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:51.940293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:51.940293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:51.940293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:51.940293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:51.940293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:51.940293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:51.940293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 01:06:51.940293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:51.940293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:51.940293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:51.940293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:51.940293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:51.940293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:51.940293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:51.940293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:51.941435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:51.941435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:51.941435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:51.941435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:51.941435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:51.941435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:51.941435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:51.941435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:51.941435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:51.941435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:51.941435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:51.941435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:51.941435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:51.941435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:51.941435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:51.941435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:51.942389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:51.942389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:51.942389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:51.942389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:51.942389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:51.942389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:51.942389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:51.942389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:51.942389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:51.942389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:51.942389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:51.942389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:51.942389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 01:06:51.942389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:51.946576 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 520/1027] 10.222.0.15 () {56 vars in 1105 bytes} [Tue Sep 13 01:06:51 2022] GET /identity/v3/services/image => generated 86 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:51.947872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831029e0-9570-476f-a2d9-98dea24ff8dd None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:52.075012 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831029e0-9570-476f-a2d9-98dea24ff8dd None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:52.102288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f90c59c-b691-4f42-857f-74584c9dead6 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:52.102578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f90c59c-b691-4f42-857f-74584c9dead6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:52.102888 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f90c59c-b691-4f42-857f-74584c9dead6 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:52.104223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 521/1028] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:06:52.108990 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35350818-6c13-45fc-86b6-c65fd910e137 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:52.109280 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35350818-6c13-45fc-86b6-c65fd910e137 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:52.109550 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35350818-6c13-45fc-86b6-c65fd910e137 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:52.112601 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b228878-e471-44aa-8273-0c1b994a9009 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:52.113435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b228878-e471-44aa-8273-0c1b994a9009 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:52.113925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b228878-e471-44aa-8273-0c1b994a9009 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:52.115011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b228878-e471-44aa-8273-0c1b994a9009 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:52.115789 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b228878-e471-44aa-8273-0c1b994a9009 None admin] RBAC: Adding query filter params (name=image) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:52.116266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b228878-e471-44aa-8273-0c1b994a9009 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:52.117760 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b228878-e471-44aa-8273-0c1b994a9009 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:52.122978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 508/1029] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:51 2022] GET /identity/v3/services?name=image => generated 122 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:52.128532 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b228878-e471-44aa-8273-0c1b994a9009 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:52.128677 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-35350818-6c13-45fc-86b6-c65fd910e137 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:52.129846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-35350818-6c13-45fc-86b6-c65fd910e137 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:52.138249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:52.208431 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 522/1030] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:52 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 01:06:52.236448 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b228878-e471-44aa-8273-0c1b994a9009 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:52.270715 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ba371e-d048-4c59-91cb-b78b68af73d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:52.271197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ba371e-d048-4c59-91cb-b78b68af73d0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:52.271473 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ba371e-d048-4c59-91cb-b78b68af73d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:52.271738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ba371e-d048-4c59-91cb-b78b68af73d0 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:52.272350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ba371e-d048-4c59-91cb-b78b68af73d0 None admin] RBAC: Adding query filter params (type=image) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:52.272645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ba371e-d048-4c59-91cb-b78b68af73d0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:52.273993 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ba371e-d048-4c59-91cb-b78b68af73d0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:52.283603 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 509/1031] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:52 2022] GET /identity/v3/services?type=image => generated 350 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:52.285310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35350818-6c13-45fc-86b6-c65fd910e137 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:52.349962 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:52.399186 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35350818-6c13-45fc-86b6-c65fd910e137 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:52.434533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614f9de9-8450-40ad-aa51-adf79f9d0406 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:52.435174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614f9de9-8450-40ad-aa51-adf79f9d0406 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:52.435448 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614f9de9-8450-40ad-aa51-adf79f9d0406 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:52.435715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614f9de9-8450-40ad-aa51-adf79f9d0406 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:52.436406 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614f9de9-8450-40ad-aa51-adf79f9d0406 None admin] RBAC: Adding query filter params (interface=public, service_id=ce2b803ad1384b1f91e2512a49022cf4, region_id=RegionOne) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:52.436706 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614f9de9-8450-40ad-aa51-adf79f9d0406 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:52.438309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614f9de9-8450-40ad-aa51-adf79f9d0406 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:52.448168 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 523/1032] 10.222.0.15 () {56 vars in 1251 bytes} [Tue Sep 13 01:06:52 2022] GET /identity/v3/endpoints?service_id=ce2b803ad1384b1f91e2512a49022cf4&interface=public®ion_id=RegionOne => generated 194 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:52.449111 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ba371e-d048-4c59-91cb-b78b68af73d0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:52.476005 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614f9de9-8450-40ad-aa51-adf79f9d0406 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:52.522620 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:52.556388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ba371e-d048-4c59-91cb-b78b68af73d0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:52.579567 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614f9de9-8450-40ad-aa51-adf79f9d0406 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:52.589450 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f4e9e9-62c0-4c70-92c0-837ef0fe6ab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:52.590066 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f4e9e9-62c0-4c70-92c0-837ef0fe6ab0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:52.590340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f4e9e9-62c0-4c70-92c0-837ef0fe6ab0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:52.590608 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f4e9e9-62c0-4c70-92c0-837ef0fe6ab0 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:52.591233 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f4e9e9-62c0-4c70-92c0-837ef0fe6ab0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:52.592578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f4e9e9-62c0-4c70-92c0-837ef0fe6ab0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:52.598736 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:52.599568 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 510/1033] 10.222.0.15 () {56 vars in 1087 bytes} [Tue Sep 13 01:06:52 2022] GET /identity/v3/services => generated 1957 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:06:52.628124 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e94808c-9d16-498c-bb1e-d538f599c595 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:52.628696 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e94808c-9d16-498c-bb1e-d538f599c595 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:52.628973 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e94808c-9d16-498c-bb1e-d538f599c595 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:52.629243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e94808c-9d16-498c-bb1e-d538f599c595 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:52.643902 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e94808c-9d16-498c-bb1e-d538f599c595 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:52.649021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-4e94808c-9d16-498c-bb1e-d538f599c595 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 01:06:52.649021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:52.649021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:52.649021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:52.649021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:52.649021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:52.649021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:52.649021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:52.649021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:52.649021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:52.649021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:52.649021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:52.649021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:52.649021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:52.649021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:52.649021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:52.650059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:52.650059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:52.650059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:52.650059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:52.650059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:52.650059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:52.650059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:52.650059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:52.650059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:52.650059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:52.650059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:52.650059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:52.650059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:52.650059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:52.650059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:52.650059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:52.651416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:52.651416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:52.651416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:52.651416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:52.651416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:52.651416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:52.651416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:52.651416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:52.651416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:52.651416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:52.651416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 01:06:52.651416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:52.655360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 524/1034] 10.222.0.15 () {56 vars in 1120 bytes} [Tue Sep 13 01:06:52 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:52.657232 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f4e9e9-62c0-4c70-92c0-837ef0fe6ab0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:52.767844 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f4e9e9-62c0-4c70-92c0-837ef0fe6ab0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:52.805797 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729e9336-35b5-48cf-ba63-8c5df3521234 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:52.806432 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729e9336-35b5-48cf-ba63-8c5df3521234 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:52.806714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729e9336-35b5-48cf-ba63-8c5df3521234 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:52.807025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729e9336-35b5-48cf-ba63-8c5df3521234 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:52.807712 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729e9336-35b5-48cf-ba63-8c5df3521234 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:52.808024 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729e9336-35b5-48cf-ba63-8c5df3521234 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:52.809434 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729e9336-35b5-48cf-ba63-8c5df3521234 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:52.811299 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:52.820538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 511/1035] 10.222.0.15 () {56 vars in 1115 bytes} [Tue Sep 13 01:06:52 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:52.826507 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e94808c-9d16-498c-bb1e-d538f599c595 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:52.929917 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e94808c-9d16-498c-bb1e-d538f599c595 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:52.964476 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1080865-3862-40df-b3b1-5838afa95d7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:52.965116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1080865-3862-40df-b3b1-5838afa95d7f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:52.965399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1080865-3862-40df-b3b1-5838afa95d7f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:52.965672 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1080865-3862-40df-b3b1-5838afa95d7f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:52.979246 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1080865-3862-40df-b3b1-5838afa95d7f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:52.980699 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1080865-3862-40df-b3b1-5838afa95d7f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:52.995369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-f1080865-3862-40df-b3b1-5838afa95d7f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:52.995369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:52.995369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:52.995369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:52.995369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:52.995369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:52.995369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:52.995369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:52.995369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:52.995369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:52.995369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:52.995369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:52.995369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:52.995369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:52.995369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:52.995369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:52.996324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:52.996324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:52.996324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:52.996324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:52.996324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:52.996324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:52.996324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:52.996324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:52.996324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:52.996324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:52.996324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:52.996324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:52.996324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:52.996324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:52.996324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:52.996324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:52.996324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:52.997251 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:52.997251 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:52.997251 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:52.997251 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 525/1036] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:53.003224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729e9336-35b5-48cf-ba63-8c5df3521234 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:53.103196 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729e9336-35b5-48cf-ba63-8c5df3521234 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:53.136924 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b699b8e2-789a-46c8-9e4c-848f08133b58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:53.137536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b699b8e2-789a-46c8-9e4c-848f08133b58 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:53.137816 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b699b8e2-789a-46c8-9e4c-848f08133b58 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:53.138087 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b699b8e2-789a-46c8-9e4c-848f08133b58 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:53.138725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b699b8e2-789a-46c8-9e4c-848f08133b58 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:53.139056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b699b8e2-789a-46c8-9e4c-848f08133b58 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:53.140414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b699b8e2-789a-46c8-9e4c-848f08133b58 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:53.152485 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 512/1037] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:53.157822 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1080865-3862-40df-b3b1-5838afa95d7f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:53.190709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:53.263332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1080865-3862-40df-b3b1-5838afa95d7f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:53.297026 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7848c814-0eae-4330-bdb5-a79bc669e468 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:53.297656 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7848c814-0eae-4330-bdb5-a79bc669e468 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:53.297933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7848c814-0eae-4330-bdb5-a79bc669e468 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:53.298201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7848c814-0eae-4330-bdb5-a79bc669e468 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:53.306344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7848c814-0eae-4330-bdb5-a79bc669e468 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:53.307801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7848c814-0eae-4330-bdb5-a79bc669e468 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:53.319521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-7848c814-0eae-4330-bdb5-a79bc669e468 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 01:06:53.319521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:53.319521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:53.319521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:53.319521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:53.319521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:53.319521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:53.319521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:53.319521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:53.319521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:53.319521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:53.319521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:53.319521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:53.319521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:53.319521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:53.319521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:53.320495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:53.320495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:53.320495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:53.320495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:53.320495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:53.320495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:53.320495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:53.320495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:53.320495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:53.320495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:53.320495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:53.320495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:53.320495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:53.320495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:53.320495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:53.320495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:53.320495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:53.320495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:53.321458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:53.321458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:53.321458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:53.321458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:53.321458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:53.321458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:53.321458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:53.321458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:53.321458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 01:06:53.321458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:53.323173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 526/1038] 10.222.0.15 () {56 vars in 1093 bytes} [Tue Sep 13 01:06:53 2022] GET /identity/v3/users/nova => generated 82 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:53.326327 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b699b8e2-789a-46c8-9e4c-848f08133b58 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:53.382884 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:53.432382 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b699b8e2-789a-46c8-9e4c-848f08133b58 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:53.472714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1b2a50-3742-49b1-90bd-021eab09c1c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:53.472714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1b2a50-3742-49b1-90bd-021eab09c1c3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:53.472714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1b2a50-3742-49b1-90bd-021eab09c1c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:53.472714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1b2a50-3742-49b1-90bd-021eab09c1c3 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:53.473366 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1b2a50-3742-49b1-90bd-021eab09c1c3 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:53.473676 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1b2a50-3742-49b1-90bd-021eab09c1c3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:53.475160 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1b2a50-3742-49b1-90bd-021eab09c1c3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:53.500747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 513/1039] 10.222.0.15 () {56 vars in 1133 bytes} [Tue Sep 13 01:06:53 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:53.502153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7848c814-0eae-4330-bdb5-a79bc669e468 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:53.591657 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:53.602091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7848c814-0eae-4330-bdb5-a79bc669e468 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:53.636258 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9efefb-ef5b-43e1-ab4f-91519e0c38f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:53.636867 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9efefb-ef5b-43e1-ab4f-91519e0c38f1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:53.637146 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9efefb-ef5b-43e1-ab4f-91519e0c38f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:53.637419 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9efefb-ef5b-43e1-ab4f-91519e0c38f1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:53.703881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9efefb-ef5b-43e1-ab4f-91519e0c38f1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:53.705317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9efefb-ef5b-43e1-ab4f-91519e0c38f1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:53.793311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:53.806181 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 527/1040] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:06:53 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 13 01:06:53.811554 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1b2a50-3742-49b1-90bd-021eab09c1c3 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:53.915744 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1b2a50-3742-49b1-90bd-021eab09c1c3 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:53.933138 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:53.953927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f34a553-a87a-4b53-8d47-8b65c473d07d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:53.954553 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f34a553-a87a-4b53-8d47-8b65c473d07d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:53.954825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f34a553-a87a-4b53-8d47-8b65c473d07d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:53.955118 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f34a553-a87a-4b53-8d47-8b65c473d07d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:53.965544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f34a553-a87a-4b53-8d47-8b65c473d07d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:53.967012 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f34a553-a87a-4b53-8d47-8b65c473d07d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:53.983019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-4f34a553-a87a-4b53-8d47-8b65c473d07d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:53.983019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:53.983019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:53.983019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:53.983019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:53.983019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:53.983019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:53.983019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:53.983019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:53.983019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:53.983019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:53.983019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:53.983019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:53.983019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:53.983019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:53.983019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:53.984039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:53.984039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:53.984039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:53.984039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:53.984039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:53.984039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:53.984039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:53.984039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:53.984039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:53.984039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:53.984039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:53.984039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:53.984039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:53.984039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:53.984039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:53.984039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:53.984039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:53.984926 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:53.984926 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:53.984926 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:53.984926 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 514/1041] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:53.991224 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9efefb-ef5b-43e1-ab4f-91519e0c38f1 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:54.091859 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9efefb-ef5b-43e1-ab4f-91519e0c38f1 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:54.099134 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0b2187-715b-4c30-b4ff-5385f189e00e None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:54.099423 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0b2187-715b-4c30-b4ff-5385f189e00e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:54.099690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0b2187-715b-4c30-b4ff-5385f189e00e None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:54.101161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 515/1042] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:54.105862 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765637c5-823d-4999-bea3-3b2eafe2d09d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:54.106144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765637c5-823d-4999-bea3-3b2eafe2d09d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:54.106416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765637c5-823d-4999-bea3-3b2eafe2d09d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:54.125199 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-765637c5-823d-4999-bea3-3b2eafe2d09d None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:54.126379 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-765637c5-823d-4999-bea3-3b2eafe2d09d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:54.126689 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc763201-5b93-43a6-b127-8b645666a2a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:54.127323 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc763201-5b93-43a6-b127-8b645666a2a0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:54.127597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc763201-5b93-43a6-b127-8b645666a2a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:54.127862 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc763201-5b93-43a6-b127-8b645666a2a0 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:54.128487 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc763201-5b93-43a6-b127-8b645666a2a0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:54.128787 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc763201-5b93-43a6-b127-8b645666a2a0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:54.130150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc763201-5b93-43a6-b127-8b645666a2a0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:54.141813 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 528/1043] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:54.146995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc763201-5b93-43a6-b127-8b645666a2a0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:54.199091 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 516/1044] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:54 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 01:06:54.200286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:54.251374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc763201-5b93-43a6-b127-8b645666a2a0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:54.290000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b044b388-c85e-4db4-b913-89865f42635a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:54.290760 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b044b388-c85e-4db4-b913-89865f42635a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:54.290984 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b044b388-c85e-4db4-b913-89865f42635a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:54.291246 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b044b388-c85e-4db4-b913-89865f42635a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:54.299125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b044b388-c85e-4db4-b913-89865f42635a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:54.300718 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b044b388-c85e-4db4-b913-89865f42635a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:54.313394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-b044b388-c85e-4db4-b913-89865f42635a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:54.313394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:54.313394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:54.313394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:54.313394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:54.313394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:54.313394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:54.313394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:54.313394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:54.313394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:54.313394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:54.313394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:54.313394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:54.313394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:54.313394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:54.314332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:54.314332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:54.314332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:54.314332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:54.314332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:54.314332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:54.314332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:54.314332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:54.314332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:54.314332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:54.314332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:54.314332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:54.314332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:54.314332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:54.314332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:54.314332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:54.314332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:54.319838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:54.319838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:54.319838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:54.319838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:54.319838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:54.319838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:54.319838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:54.319838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:54.319838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 529/1045] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:54.322904 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765637c5-823d-4999-bea3-3b2eafe2d09d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:54.423158 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:54.430646 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765637c5-823d-4999-bea3-3b2eafe2d09d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:54.466738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c1d4e3-30e7-494d-a407-aec43208398e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:54.467421 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c1d4e3-30e7-494d-a407-aec43208398e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:54.467666 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c1d4e3-30e7-494d-a407-aec43208398e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:54.467936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c1d4e3-30e7-494d-a407-aec43208398e None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:54.468384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b044b388-c85e-4db4-b913-89865f42635a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:54.468621 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c1d4e3-30e7-494d-a407-aec43208398e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:54.468924 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c1d4e3-30e7-494d-a407-aec43208398e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:54.470348 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c1d4e3-30e7-494d-a407-aec43208398e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:54.482439 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 517/1046] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:54.488122 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c1d4e3-30e7-494d-a407-aec43208398e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:54.587942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b044b388-c85e-4db4-b913-89865f42635a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:54.594998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c1d4e3-30e7-494d-a407-aec43208398e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:54.599310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:54.626417 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58868587-db6a-471a-8450-784718930944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:54.627157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58868587-db6a-471a-8450-784718930944 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:54.627437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58868587-db6a-471a-8450-784718930944 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:54.627710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58868587-db6a-471a-8450-784718930944 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:54.639177 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58868587-db6a-471a-8450-784718930944 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:54.643304 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb35a819-65ed-4be9-adbc-d1e99de7a4a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:54.643918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-58868587-db6a-471a-8450-784718930944 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:06:54.643918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:54.643918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:54.643918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:54.643918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:54.643918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:54.643918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:54.643918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:54.643918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:54.643918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:54.643918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:54.643918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:54.643918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:54.643918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:54.643918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:54.643918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:54.644841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:54.644841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:54.644841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:54.644841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:54.644841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:54.644841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:54.644841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:54.644841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:54.644841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:54.644841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:54.644841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:54.644841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:54.644841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:54.644841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:54.644841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:54.644841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:54.646228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:54.646228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:54.646228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:54.646228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:54.646228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:54.646228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:54.646228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:54.646228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:54.646228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:54.646228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:54.646228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:06:54.646228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:54.646891 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb35a819-65ed-4be9-adbc-d1e99de7a4a6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:54.646891 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb35a819-65ed-4be9-adbc-d1e99de7a4a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:54.646891 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb35a819-65ed-4be9-adbc-d1e99de7a4a6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:54.647098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 530/1047] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:06:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:54.650602 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb35a819-65ed-4be9-adbc-d1e99de7a4a6 None admin] RBAC: Adding query filter params (role.id=11f2e919cae541428f8c564b7b67eea0, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=2f2da4d2d3604d16b58dc25b8c17c49a) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:54.650965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb35a819-65ed-4be9-adbc-d1e99de7a4a6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:54.651385 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58868587-db6a-471a-8450-784718930944 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:54.652611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb35a819-65ed-4be9-adbc-d1e99de7a4a6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:54.663265 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 518/1048] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:06:54 2022] GET /identity/v3/role_assignments?user.id=2f2da4d2d3604d16b58dc25b8c17c49a&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=11f2e919cae541428f8c564b7b67eea0 => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:54.772449 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58868587-db6a-471a-8450-784718930944 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:54.805066 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f53d1c5-ad16-4a1b-a253-0d819b82f8a2 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:54.805342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f53d1c5-ad16-4a1b-a253-0d819b82f8a2 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:54.805655 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f53d1c5-ad16-4a1b-a253-0d819b82f8a2 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:54.807177 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 519/1049] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:06:54.808866 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c32d202-510c-4ff1-9433-ae1495a6bc23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:54.809486 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c32d202-510c-4ff1-9433-ae1495a6bc23 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:54.809764 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c32d202-510c-4ff1-9433-ae1495a6bc23 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:54.810033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c32d202-510c-4ff1-9433-ae1495a6bc23 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:54.810705 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c32d202-510c-4ff1-9433-ae1495a6bc23 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:54.811039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c32d202-510c-4ff1-9433-ae1495a6bc23 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:54.812504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c32d202-510c-4ff1-9433-ae1495a6bc23 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:54.813005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791b224c-509f-4ec4-9f9e-150996b4c216 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:54.813294 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791b224c-509f-4ec4-9f9e-150996b4c216 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:54.813568 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791b224c-509f-4ec4-9f9e-150996b4c216 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:54.823257 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 531/1050] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:06:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:54.829220 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c32d202-510c-4ff1-9433-ae1495a6bc23 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:54.835133 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-791b224c-509f-4ec4-9f9e-150996b4c216 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:54.836572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-791b224c-509f-4ec4-9f9e-150996b4c216 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:54.857151 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:54.870668 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:54.914273 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 520/1051] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:54 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:54.953744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c32d202-510c-4ff1-9433-ae1495a6bc23 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:54.989120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e1fed8-482e-4182-a2bd-83d8f71a28f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:54.989754 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e1fed8-482e-4182-a2bd-83d8f71a28f4 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:54.990033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e1fed8-482e-4182-a2bd-83d8f71a28f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:54.990306 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e1fed8-482e-4182-a2bd-83d8f71a28f4 None admin] PATH_INFO: `/v3/users/42fe3f2b90dc4d33b1f0e1e77da63224` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:55.003923 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e1fed8-482e-4182-a2bd-83d8f71a28f4 None admin] RBAC: Authorizing `identity:get_user(user_id=42fe3f2b90dc4d33b1f0e1e77da63224)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:55.005878 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e1fed8-482e-4182-a2bd-83d8f71a28f4 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:55.007709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 532/1052] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:06:54 2022] GET /identity/v3/users/42fe3f2b90dc4d33b1f0e1e77da63224 => generated 285 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:55.013270 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791b224c-509f-4ec4-9f9e-150996b4c216 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:55.115019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791b224c-509f-4ec4-9f9e-150996b4c216 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:55.151199 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ee8860-3487-49ae-991b-91716f6585b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:55.151807 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ee8860-3487-49ae-991b-91716f6585b6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:55.152085 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ee8860-3487-49ae-991b-91716f6585b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:55.152353 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ee8860-3487-49ae-991b-91716f6585b6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:55.170949 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:55.184192 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e1fed8-482e-4182-a2bd-83d8f71a28f4 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:55.227436 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ee8860-3487-49ae-991b-91716f6585b6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:55.228884 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ee8860-3487-49ae-991b-91716f6585b6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:55.284176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:55.296168 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e1fed8-482e-4182-a2bd-83d8f71a28f4 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:55.333548 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b5bef3-6444-42c1-a48d-6a441d06f34b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:55.334176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b5bef3-6444-42c1-a48d-6a441d06f34b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:55.334447 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b5bef3-6444-42c1-a48d-6a441d06f34b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:55.334712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b5bef3-6444-42c1-a48d-6a441d06f34b None admin] PATH_INFO: `/v3/services/image` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:55.335305 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b5bef3-6444-42c1-a48d-6a441d06f34b None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:55.336648 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b5bef3-6444-42c1-a48d-6a441d06f34b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:55.338104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 521/1053] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:06:55 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 13 01:06:55.343954 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ee8860-3487-49ae-991b-91716f6585b6 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:55.350919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-96b5bef3-6444-42c1-a48d-6a441d06f34b None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 01:06:55.350919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:55.350919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:06:55.350919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:06:55.350919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:06:55.350919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:06:55.350919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:06:55.350919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:55.350919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 01:06:55.350919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:55.350919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:06:55.350919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:55.350919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:55.350919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:55.350919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:55.350919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:55.350919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:55.351967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:55.351967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:55.351967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:55.351967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:55.351967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:55.351967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:55.351967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:06:55.351967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:06:55.351967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:06:55.351967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:06:55.351967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:55.351967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:55.351967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:55.351967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:55.351967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:55.351967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:55.352827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:55.352827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:55.352827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:55.352827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:55.352827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:55.352827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:55.352827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:55.352827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:55.352827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:55.352827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:55.352827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:06:55.352827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:06:55.352827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 01:06:55.352827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:55.355169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 533/1054] 10.222.0.15 () {56 vars in 1105 bytes} [Tue Sep 13 01:06:55 2022] GET /identity/v3/services/image => generated 86 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:55.358388 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b5bef3-6444-42c1-a48d-6a441d06f34b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:55.397307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:55.452795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ee8860-3487-49ae-991b-91716f6585b6 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:55.472311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b5bef3-6444-42c1-a48d-6a441d06f34b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:55.492363 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dece59-18eb-4042-8d26-ba757a3b50d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:55.493381 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dece59-18eb-4042-8d26-ba757a3b50d5 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:55.493846 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dece59-18eb-4042-8d26-ba757a3b50d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:55.494271 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dece59-18eb-4042-8d26-ba757a3b50d5 None admin] PATH_INFO: `/v3/projects/656038fa03254d489978b4958195a414` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:55.495545 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dece59-18eb-4042-8d26-ba757a3b50d5 None admin] RBAC: Authorizing `identity:get_project(project_id=656038fa03254d489978b4958195a414)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:55.497325 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dece59-18eb-4042-8d26-ba757a3b50d5 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:55.499485 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 522/1055] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:06:55 2022] GET /identity/v3/projects/656038fa03254d489978b4958195a414 => generated 311 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:55.505706 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dece59-18eb-4042-8d26-ba757a3b50d5 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:55.514896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06793ed8-b47c-44c7-86f4-9b5ad583e2e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:55.515514 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06793ed8-b47c-44c7-86f4-9b5ad583e2e1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:55.515795 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06793ed8-b47c-44c7-86f4-9b5ad583e2e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:55.516236 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06793ed8-b47c-44c7-86f4-9b5ad583e2e1 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:55.516871 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06793ed8-b47c-44c7-86f4-9b5ad583e2e1 None admin] RBAC: Adding query filter params (name=image) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:55.517174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06793ed8-b47c-44c7-86f4-9b5ad583e2e1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:55.518533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06793ed8-b47c-44c7-86f4-9b5ad583e2e1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:55.524773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 534/1056] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:55 2022] GET /identity/v3/services?name=image => generated 122 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:55.530589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06793ed8-b47c-44c7-86f4-9b5ad583e2e1 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:55.551903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:55.613814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dece59-18eb-4042-8d26-ba757a3b50d5 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:55.632535 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:55.642698 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06793ed8-b47c-44c7-86f4-9b5ad583e2e1 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:55.650469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eae4b85-61ed-41f0-a9e6-6ba44d8698e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:55.651097 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eae4b85-61ed-41f0-a9e6-6ba44d8698e9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:55.651376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eae4b85-61ed-41f0-a9e6-6ba44d8698e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:55.651644 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eae4b85-61ed-41f0-a9e6-6ba44d8698e9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:55.652354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eae4b85-61ed-41f0-a9e6-6ba44d8698e9 None admin] RBAC: Adding query filter params (role.id=66a36c858aab40688080eed6ceace3a0, scope.project.id=656038fa03254d489978b4958195a414, user.id=42fe3f2b90dc4d33b1f0e1e77da63224) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:55.652654 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eae4b85-61ed-41f0-a9e6-6ba44d8698e9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:55.654062 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eae4b85-61ed-41f0-a9e6-6ba44d8698e9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:55.660230 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 523/1057] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:06:55 2022] GET /identity/v3/role_assignments?user.id=42fe3f2b90dc4d33b1f0e1e77da63224&scope.project.id=656038fa03254d489978b4958195a414&role.id=66a36c858aab40688080eed6ceace3a0 => generated 259 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:55.681763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea286c68-9627-4d25-ad2c-666f5e510a95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:55.682394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea286c68-9627-4d25-ad2c-666f5e510a95 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:55.682720 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea286c68-9627-4d25-ad2c-666f5e510a95 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:55.683023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea286c68-9627-4d25-ad2c-666f5e510a95 None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:55.683663 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea286c68-9627-4d25-ad2c-666f5e510a95 None admin] RBAC: Adding query filter params (type=image) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:55.683961 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea286c68-9627-4d25-ad2c-666f5e510a95 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:55.685348 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea286c68-9627-4d25-ad2c-666f5e510a95 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:55.691582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 535/1058] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:55 2022] GET /identity/v3/services?type=image => generated 350 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:55.697816 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eae4b85-61ed-41f0-a9e6-6ba44d8698e9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:55.806776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eae4b85-61ed-41f0-a9e6-6ba44d8698e9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:55.844499 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad15478-743a-4b71-a084-4f0b2af414a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:55.845112 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad15478-743a-4b71-a084-4f0b2af414a9 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:55.845393 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad15478-743a-4b71-a084-4f0b2af414a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:55.845665 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad15478-743a-4b71-a084-4f0b2af414a9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:55.846341 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad15478-743a-4b71-a084-4f0b2af414a9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:55.847781 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad15478-743a-4b71-a084-4f0b2af414a9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:55.871004 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 524/1059] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:06:55 2022] POST /identity/v3/endpoints => generated 327 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:06:56.816373 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5483836-e434-4fd2-b0fd-e634f406b090 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:56.816995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5483836-e434-4fd2-b0fd-e634f406b090 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:56.816995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5483836-e434-4fd2-b0fd-e634f406b090 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:56.818579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 536/1060] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:06:56.823908 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba25b3a1-8ce6-47d8-87e1-7a5fd8553ec3 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:56.824340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba25b3a1-8ce6-47d8-87e1-7a5fd8553ec3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:56.824553 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba25b3a1-8ce6-47d8-87e1-7a5fd8553ec3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:56.847764 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ba25b3a1-8ce6-47d8-87e1-7a5fd8553ec3 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:56.849054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ba25b3a1-8ce6-47d8-87e1-7a5fd8553ec3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:56.888625 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:56.940645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 525/1061] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:56 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:06:57.204777 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5483836-e434-4fd2-b0fd-e634f406b090 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:57.287184 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:57.307663 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5483836-e434-4fd2-b0fd-e634f406b090 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:57.340741 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd1c826-9fbe-4c74-813d-cb308f708adc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:57.341388 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd1c826-9fbe-4c74-813d-cb308f708adc None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:57.341659 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd1c826-9fbe-4c74-813d-cb308f708adc None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:57.341925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd1c826-9fbe-4c74-813d-cb308f708adc None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:57.353883 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd1c826-9fbe-4c74-813d-cb308f708adc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:57.358610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-6cd1c826-9fbe-4c74-813d-cb308f708adc None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 01:06:57.358610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:57.358610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:57.358610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:57.358610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:57.358610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:57.358610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:57.358610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:57.358610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:57.358610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:57.358610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:57.358610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:57.358610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:57.358610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:57.358610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:57.358610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:57.359585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:57.359585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:57.359585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:57.359585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:57.359585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:57.359585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:57.359585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:57.359585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:57.359585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:57.359585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:57.359585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:57.359585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:57.359585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:57.359585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:57.359585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:57.359585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:57.360457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:57.360457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:57.360457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:57.360457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:57.360457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:57.360457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:57.360457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:57.360457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:57.360457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:57.360457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:57.360457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 01:06:57.360457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:57.362931 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 537/1062] 10.222.0.15 () {56 vars in 1120 bytes} [Tue Sep 13 01:06:57 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:57.366621 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba25b3a1-8ce6-47d8-87e1-7a5fd8553ec3 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:57.484509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba25b3a1-8ce6-47d8-87e1-7a5fd8553ec3 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:57.518169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2113bb79-c392-4e16-b0c8-23922f9bdf35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:57.518787 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2113bb79-c392-4e16-b0c8-23922f9bdf35 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:57.519094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2113bb79-c392-4e16-b0c8-23922f9bdf35 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:57.519354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2113bb79-c392-4e16-b0c8-23922f9bdf35 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:57.520541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2113bb79-c392-4e16-b0c8-23922f9bdf35 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:57.520845 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2113bb79-c392-4e16-b0c8-23922f9bdf35 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:57.522203 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2113bb79-c392-4e16-b0c8-23922f9bdf35 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:57.534962 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 526/1063] 10.222.0.15 () {56 vars in 1115 bytes} [Tue Sep 13 01:06:57 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:57.537186 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd1c826-9fbe-4c74-813d-cb308f708adc None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:57.637885 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd1c826-9fbe-4c74-813d-cb308f708adc None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:57.671166 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42463be-6c2f-4fd7-9a90-088224c5be74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:57.671822 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42463be-6c2f-4fd7-9a90-088224c5be74 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:57.672212 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42463be-6c2f-4fd7-9a90-088224c5be74 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:57.672487 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42463be-6c2f-4fd7-9a90-088224c5be74 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:57.685296 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42463be-6c2f-4fd7-9a90-088224c5be74 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:57.686762 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42463be-6c2f-4fd7-9a90-088224c5be74 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:57.689164 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:57.705586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-f42463be-6c2f-4fd7-9a90-088224c5be74 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:57.705586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:57.705586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:57.705586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:57.705586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:57.705586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:57.705586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:57.705586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:57.705586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:57.705586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:57.705586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:57.705586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:57.705586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:57.705586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:57.705586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:57.705586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:57.706615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:57.706615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:57.706615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:57.706615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:57.706615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:57.706615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:57.706615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:57.706615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:57.706615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:57.706615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:57.706615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:57.706615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:57.706615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:57.706615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:57.706615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:57.706615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:57.706615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:57.707623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:57.707623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:57.707623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:57.707623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 538/1064] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:57.712679 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2113bb79-c392-4e16-b0c8-23922f9bdf35 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:57.741105 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:57.794216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e4d2c9-8f08-4073-923e-d8258516bfe7 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:57.794500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e4d2c9-8f08-4073-923e-d8258516bfe7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:57.794772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e4d2c9-8f08-4073-923e-d8258516bfe7 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:57.796128 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 539/1065] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:06:57.800914 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0155267-5625-4688-958f-04ff9166a8ec None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:57.801734 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0155267-5625-4688-958f-04ff9166a8ec None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:57.802008 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0155267-5625-4688-958f-04ff9166a8ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:57.817543 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2113bb79-c392-4e16-b0c8-23922f9bdf35 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:57.821663 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b0155267-5625-4688-958f-04ff9166a8ec None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:57.822834 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b0155267-5625-4688-958f-04ff9166a8ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:57.853321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708a2c2a-3f2a-456e-858f-5226b9e58f84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:57.854225 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708a2c2a-3f2a-456e-858f-5226b9e58f84 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:57.854505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708a2c2a-3f2a-456e-858f-5226b9e58f84 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:57.854775 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708a2c2a-3f2a-456e-858f-5226b9e58f84 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:57.855437 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708a2c2a-3f2a-456e-858f-5226b9e58f84 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:57.855736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708a2c2a-3f2a-456e-858f-5226b9e58f84 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:57.857090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708a2c2a-3f2a-456e-858f-5226b9e58f84 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:57.868896 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 527/1066] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:57.874291 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708a2c2a-3f2a-456e-858f-5226b9e58f84 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:57.893344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 540/1067] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 01:06:57.964246 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:57.981893 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708a2c2a-3f2a-456e-858f-5226b9e58f84 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:58.015766 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f5164f-6c1d-4601-b194-92ba38317d5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:58.016317 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f5164f-6c1d-4601-b194-92ba38317d5d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:58.016597 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f5164f-6c1d-4601-b194-92ba38317d5d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:58.016870 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f5164f-6c1d-4601-b194-92ba38317d5d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:58.023846 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f5164f-6c1d-4601-b194-92ba38317d5d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:58.025268 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f5164f-6c1d-4601-b194-92ba38317d5d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:58.037235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-f2f5164f-6c1d-4601-b194-92ba38317d5d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 01:06:58.037235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:58.037235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:58.037235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:58.037235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:58.037235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:58.037235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:58.037235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:58.037235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:58.037235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:58.037235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:58.037235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:58.037235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:06:58.037235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:06:58.037235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:06:58.037235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:06:58.038161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:58.038161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:58.038161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:06:58.038161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:58.038161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:06:58.038161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:06:58.038161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:58.038161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:58.038161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:58.038161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:58.038161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:58.038161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:58.038161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:58.038161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:58.038161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:58.038161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:58.038161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:58.038161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:58.039698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:58.039698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:58.039698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:06:58.039698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:06:58.039698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:06:58.039698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:06:58.039698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:06:58.039698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:06:58.039698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 01:06:58.039698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:58.039698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 528/1068] 10.222.0.15 () {56 vars in 1093 bytes} [Tue Sep 13 01:06:57 2022] GET /identity/v3/users/nova => generated 82 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:58.044303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0155267-5625-4688-958f-04ff9166a8ec None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:58.065174 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774f2f4a-802a-4d9a-8c46-86a2c335ff24 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:58.065447 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774f2f4a-802a-4d9a-8c46-86a2c335ff24 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:58.065720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774f2f4a-802a-4d9a-8c46-86a2c335ff24 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:58.067873 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 529/1069] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:06:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:06:58.072389 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae3a167-da2f-4e19-b1a7-238efc715b2a None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:58.072689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae3a167-da2f-4e19-b1a7-238efc715b2a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:58.072967 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae3a167-da2f-4e19-b1a7-238efc715b2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:58.091670 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-eae3a167-da2f-4e19-b1a7-238efc715b2a None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:06:58.092862 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-eae3a167-da2f-4e19-b1a7-238efc715b2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:06:58.155334 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0155267-5625-4688-958f-04ff9166a8ec None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:58.159707 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 530/1070] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:06:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 01:06:58.164187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae3a167-da2f-4e19-b1a7-238efc715b2a None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:58.180450 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:58.193666 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694db1a6-8623-46ca-9c61-0df6de802ac3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:58.194839 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694db1a6-8623-46ca-9c61-0df6de802ac3 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:58.195152 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694db1a6-8623-46ca-9c61-0df6de802ac3 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:58.195417 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694db1a6-8623-46ca-9c61-0df6de802ac3 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:58.196140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694db1a6-8623-46ca-9c61-0df6de802ac3 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:58.196444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694db1a6-8623-46ca-9c61-0df6de802ac3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:58.197886 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694db1a6-8623-46ca-9c61-0df6de802ac3 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:58.215159 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 541/1071] 10.222.0.15 () {56 vars in 1133 bytes} [Tue Sep 13 01:06:58 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:58.220639 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694db1a6-8623-46ca-9c61-0df6de802ac3 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:58.252546 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:58.276981 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae3a167-da2f-4e19-b1a7-238efc715b2a None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:58.313785 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4694c739-103d-47a7-83b7-b2fdf117994c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:58.314408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4694c739-103d-47a7-83b7-b2fdf117994c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:58.314690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4694c739-103d-47a7-83b7-b2fdf117994c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:58.314987 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4694c739-103d-47a7-83b7-b2fdf117994c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:58.325377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4694c739-103d-47a7-83b7-b2fdf117994c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:06:58.330108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-4694c739-103d-47a7-83b7-b2fdf117994c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:06:58.330108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:58.330108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:58.330108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:58.330108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:58.330108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:58.330108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:58.330108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:58.330108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:58.330108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:58.330108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:58.330108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:58.330108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:06:58.330108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:06:58.330108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:06:58.330108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:06:58.331374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:06:58.331374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:06:58.331374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:06:58.331374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:06:58.331374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:58.331374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:58.331374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:58.331374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:58.331374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:58.331374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:58.331374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:58.331374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:58.331374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:58.331374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:58.331374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:58.331374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:58.332957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:58.332957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:58.332957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:58.332957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:58.332957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:06:58.332957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:06:58.332957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:06:58.332957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:06:58.332957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:06:58.332957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:06:58.332957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:06:58.332957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:06:58.336865 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 531/1072] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:06:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:58.337034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694db1a6-8623-46ca-9c61-0df6de802ac3 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:58.337601 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4694c739-103d-47a7-83b7-b2fdf117994c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:58.373970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c025e7-a596-4fd7-9b58-346b324612bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:58.374596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c025e7-a596-4fd7-9b58-346b324612bc None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:58.374899 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c025e7-a596-4fd7-9b58-346b324612bc None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:58.375181 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c025e7-a596-4fd7-9b58-346b324612bc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:58.386606 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c025e7-a596-4fd7-9b58-346b324612bc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:58.388050 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c025e7-a596-4fd7-9b58-346b324612bc None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:58.404986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-82c025e7-a596-4fd7-9b58-346b324612bc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:58.404986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:58.404986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:58.404986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:58.404986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:58.404986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:58.404986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:58.404986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:58.404986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:58.404986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:58.404986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:58.404986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:58.404986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:58.404986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:58.404986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:58.404986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:58.405977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:58.405977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:58.405977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:58.405977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:58.405977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:58.405977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:58.405977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:58.405977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:58.405977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:58.405977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:58.405977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:58.405977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:58.405977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:58.405977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:58.405977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:58.405977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:58.405977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:58.407135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:58.407135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:58.407135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:58.411496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 542/1073] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:06:58.412481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c025e7-a596-4fd7-9b58-346b324612bc None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:58.446814 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:58.450802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4694c739-103d-47a7-83b7-b2fdf117994c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:58.485604 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9b91d8-ec87-4ef9-b6a1-a47d5fa99283 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:58.486208 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9b91d8-ec87-4ef9-b6a1-a47d5fa99283 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:58.486491 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9b91d8-ec87-4ef9-b6a1-a47d5fa99283 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:58.486765 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9b91d8-ec87-4ef9-b6a1-a47d5fa99283 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:58.487454 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9b91d8-ec87-4ef9-b6a1-a47d5fa99283 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:58.487762 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9b91d8-ec87-4ef9-b6a1-a47d5fa99283 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:58.489175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9b91d8-ec87-4ef9-b6a1-a47d5fa99283 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:58.491339 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:58.498505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 532/1074] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:06:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:06:58.499211 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9b91d8-ec87-4ef9-b6a1-a47d5fa99283 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:58.518123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c025e7-a596-4fd7-9b58-346b324612bc None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:58.551936 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c24df4-fa51-4c30-bada-3120c009d002 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:58.552461 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c24df4-fa51-4c30-bada-3120c009d002 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:58.552731 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c24df4-fa51-4c30-bada-3120c009d002 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:58.553005 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c24df4-fa51-4c30-bada-3120c009d002 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:58.553622 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c24df4-fa51-4c30-bada-3120c009d002 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:58.554178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c24df4-fa51-4c30-bada-3120c009d002 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:58.555288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c24df4-fa51-4c30-bada-3120c009d002 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:58.567486 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 543/1075] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:58.568254 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c24df4-fa51-4c30-bada-3120c009d002 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:58.602033 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9b91d8-ec87-4ef9-b6a1-a47d5fa99283 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:58.635502 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ecaf4bf-68bf-4aaa-895d-2a28f5a6bb9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:58.636189 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecaf4bf-68bf-4aaa-895d-2a28f5a6bb9c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:58.636466 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecaf4bf-68bf-4aaa-895d-2a28f5a6bb9c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:58.636735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecaf4bf-68bf-4aaa-895d-2a28f5a6bb9c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:58.658677 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:58.666347 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c24df4-fa51-4c30-bada-3120c009d002 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:58.699543 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:58.703315 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ecaf4bf-68bf-4aaa-895d-2a28f5a6bb9c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:58.703476 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7d1d64-eb0e-4622-9814-35ac0b0c5f4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:58.703936 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7d1d64-eb0e-4622-9814-35ac0b0c5f4c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:58.704210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7d1d64-eb0e-4622-9814-35ac0b0c5f4c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:58.704478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7d1d64-eb0e-4622-9814-35ac0b0c5f4c None admin] PATH_INFO: `/v3/users/42fe3f2b90dc4d33b1f0e1e77da63224` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:58.704969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ecaf4bf-68bf-4aaa-895d-2a28f5a6bb9c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:58.705588 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7d1d64-eb0e-4622-9814-35ac0b0c5f4c None admin] RBAC: Authorizing `identity:get_user(user_id=42fe3f2b90dc4d33b1f0e1e77da63224)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:58.707178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7d1d64-eb0e-4622-9814-35ac0b0c5f4c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:58.709562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 544/1076] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:06:58 2022] GET /identity/v3/users/42fe3f2b90dc4d33b1f0e1e77da63224 => generated 285 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:58.710310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7d1d64-eb0e-4622-9814-35ac0b0c5f4c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:58.802287 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 533/1077] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:06:58 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 01:06:58.803053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ecaf4bf-68bf-4aaa-895d-2a28f5a6bb9c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:58.807568 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7d1d64-eb0e-4622-9814-35ac0b0c5f4c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:58.842424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7eb1c5-e1a2-4743-a6ad-4a0e28204d3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:58.843086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7eb1c5-e1a2-4743-a6ad-4a0e28204d3c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:58.843363 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7eb1c5-e1a2-4743-a6ad-4a0e28204d3c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:58.843636 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7eb1c5-e1a2-4743-a6ad-4a0e28204d3c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:58.850742 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7eb1c5-e1a2-4743-a6ad-4a0e28204d3c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:58.852204 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7eb1c5-e1a2-4743-a6ad-4a0e28204d3c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:58.863248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-ff7eb1c5-e1a2-4743-a6ad-4a0e28204d3c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:58.863248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:58.863248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:58.863248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:58.863248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:58.863248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:58.863248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:58.863248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:58.863248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:58.863248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:58.863248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:58.863248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:58.863248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:06:58.863248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:06:58.863248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:06:58.864278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:06:58.864278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:58.864278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:58.864278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:58.864278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:58.864278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:58.864278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:58.864278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:58.864278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:58.864278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:58.864278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:58.864278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:58.864278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:58.864278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:58.864278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:58.864278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:58.864278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:58.865172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:06:58.865172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:06:58.865172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:06:58.865172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:06:58.865172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:06:58.865172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:06:58.865172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:06:58.865172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:58.865172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 545/1078] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:06:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:58.865172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7eb1c5-e1a2-4743-a6ad-4a0e28204d3c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:58.892253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:58.901767 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:58.905380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ecaf4bf-68bf-4aaa-895d-2a28f5a6bb9c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:58.940006 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200075d8-3482-423f-8807-15a8bf494f0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:58.940619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200075d8-3482-423f-8807-15a8bf494f0c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:58.940894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200075d8-3482-423f-8807-15a8bf494f0c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:58.941165 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200075d8-3482-423f-8807-15a8bf494f0c None admin] PATH_INFO: `/v3/projects/656038fa03254d489978b4958195a414` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:58.942273 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200075d8-3482-423f-8807-15a8bf494f0c None admin] RBAC: Authorizing `identity:get_project(project_id=656038fa03254d489978b4958195a414)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:58.943784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200075d8-3482-423f-8807-15a8bf494f0c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:58.945389 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 534/1079] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:06:58 2022] GET /identity/v3/projects/656038fa03254d489978b4958195a414 => generated 311 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:58.946129 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200075d8-3482-423f-8807-15a8bf494f0c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:58.973889 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7eb1c5-e1a2-4743-a6ad-4a0e28204d3c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:59.006184 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd63ce3-2d4b-40d4-b02c-a343cb01d7bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:59.006800 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd63ce3-2d4b-40d4-b02c-a343cb01d7bc None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:59.007100 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd63ce3-2d4b-40d4-b02c-a343cb01d7bc None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:59.007375 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd63ce3-2d4b-40d4-b02c-a343cb01d7bc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:59.017582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd63ce3-2d4b-40d4-b02c-a343cb01d7bc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:59.019052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd63ce3-2d4b-40d4-b02c-a343cb01d7bc None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:59.033205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-acd63ce3-2d4b-40d4-b02c-a343cb01d7bc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:59.033205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:06:59.033205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:06:59.033205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:06:59.033205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:06:59.033205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:06:59.033205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:06:59.033205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:06:59.033205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:06:59.033205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:06:59.033205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:06:59.033205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:06:59.033205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:06:59.033205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:06:59.033205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:06:59.033205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:06:59.034219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:06:59.034219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:06:59.034219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:06:59.034219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:06:59.034219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:06:59.034219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:06:59.034219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:06:59.034219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:06:59.034219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:06:59.034219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:06:59.034219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:06:59.034219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:06:59.034219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:06:59.034219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:06:59.034219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:06:59.034219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:06:59.034219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:06:59.035191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:06:59.035191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:06:59.035191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:06:59.035191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 546/1080] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:06:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:06:59.035191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd63ce3-2d4b-40d4-b02c-a343cb01d7bc None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:59.044655 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200075d8-3482-423f-8807-15a8bf494f0c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:59.079302 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839afe3a-7462-41f1-af55-828bc5888857 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:59.079840 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839afe3a-7462-41f1-af55-828bc5888857 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:59.080124 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839afe3a-7462-41f1-af55-828bc5888857 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:59.080397 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839afe3a-7462-41f1-af55-828bc5888857 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:59.081081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839afe3a-7462-41f1-af55-828bc5888857 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:59.081387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839afe3a-7462-41f1-af55-828bc5888857 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:59.083411 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839afe3a-7462-41f1-af55-828bc5888857 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:59.094534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 535/1081] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:06:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:59.095344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839afe3a-7462-41f1-af55-828bc5888857 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:59.113665 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:59.122222 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:59.138638 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd63ce3-2d4b-40d4-b02c-a343cb01d7bc None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:59.172005 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08122837-c1db-4c03-85f3-7be53747c9a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:59.172650 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08122837-c1db-4c03-85f3-7be53747c9a1 None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:59.172925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08122837-c1db-4c03-85f3-7be53747c9a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:59.173195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08122837-c1db-4c03-85f3-7be53747c9a1 None admin] PATH_INFO: `/v3/projects/656038fa03254d489978b4958195a414/users/42fe3f2b90dc4d33b1f0e1e77da63224/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:59.175108 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08122837-c1db-4c03-85f3-7be53747c9a1 None admin] RBAC: Authorizing `identity:create_grant(project_id=656038fa03254d489978b4958195a414, user_id=42fe3f2b90dc4d33b1f0e1e77da63224, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:59.176560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08122837-c1db-4c03-85f3-7be53747c9a1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:59.190159 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 547/1082] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:06:59 2022] PUT /identity/v3/projects/656038fa03254d489978b4958195a414/users/42fe3f2b90dc4d33b1f0e1e77da63224/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:06:59.190947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08122837-c1db-4c03-85f3-7be53747c9a1 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:06:59.202511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839afe3a-7462-41f1-af55-828bc5888857 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:59.248242 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab921a2-1f64-4d6c-ad55-31e637f51e1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:59.248943 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab921a2-1f64-4d6c-ad55-31e637f51e1c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:59.249223 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab921a2-1f64-4d6c-ad55-31e637f51e1c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:59.249539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab921a2-1f64-4d6c-ad55-31e637f51e1c None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:59.250961 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab921a2-1f64-4d6c-ad55-31e637f51e1c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:06:59.251268 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab921a2-1f64-4d6c-ad55-31e637f51e1c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:59.253328 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab921a2-1f64-4d6c-ad55-31e637f51e1c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:59.268184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 536/1083] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:06:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:06:59.315166 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08122837-c1db-4c03-85f3-7be53747c9a1 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:06:59.355530 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae90b1d7-f7be-4929-81f0-4ebe4c0150d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:06:59.356136 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae90b1d7-f7be-4929-81f0-4ebe4c0150d3 None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:06:59.356413 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae90b1d7-f7be-4929-81f0-4ebe4c0150d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:06:59.356690 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae90b1d7-f7be-4929-81f0-4ebe4c0150d3 None admin] PATH_INFO: `/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/2f2da4d2d3604d16b58dc25b8c17c49a/roles/11f2e919cae541428f8c564b7b67eea0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:06:59.359084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae90b1d7-f7be-4929-81f0-4ebe4c0150d3 None admin] RBAC: Authorizing `identity:create_grant(project_id=e5c1519fdae742f4a25eb7cd5476a8cb, user_id=2f2da4d2d3604d16b58dc25b8c17c49a, role_id=11f2e919cae541428f8c564b7b67eea0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:06:59.360561 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae90b1d7-f7be-4929-81f0-4ebe4c0150d3 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:06:59.363098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:06:59.378794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 548/1084] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:06:59 2022] PUT /identity/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/2f2da4d2d3604d16b58dc25b8c17c49a/roles/11f2e919cae541428f8c564b7b67eea0 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:07:01.367103 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0b22fe-c96c-4d4f-b622-28adf4e40c7d None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:01.367635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0b22fe-c96c-4d4f-b622-28adf4e40c7d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:01.367635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0b22fe-c96c-4d4f-b622-28adf4e40c7d None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:01.368983 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 537/1085] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:07:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:07:01.374147 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a93091c-8d37-4e9c-9fa6-084a68e43cb1 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:01.374536 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a93091c-8d37-4e9c-9fa6-084a68e43cb1 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:01.374702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a93091c-8d37-4e9c-9fa6-084a68e43cb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:01.392600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-6a93091c-8d37-4e9c-9fa6-084a68e43cb1 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:07:01.393770 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6a93091c-8d37-4e9c-9fa6-084a68e43cb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:07:01.469599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 549/1086] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:07:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:07:01.534948 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3118e327-4499-4323-96cc-23c78f693d2a None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:01.535267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3118e327-4499-4323-96cc-23c78f693d2a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:01.535596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3118e327-4499-4323-96cc-23c78f693d2a None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:01.537824 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 538/1087] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:07:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:07:01.543092 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edac26da-35cc-4f05-b9e1-b1701dc4352a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:01.543375 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edac26da-35cc-4f05-b9e1-b1701dc4352a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:01.543670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edac26da-35cc-4f05-b9e1-b1701dc4352a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:01.561868 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-edac26da-35cc-4f05-b9e1-b1701dc4352a None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:07:01.563319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-edac26da-35cc-4f05-b9e1-b1701dc4352a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:07:01.627361 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 550/1088] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:07:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:07:01.740464 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3118e327-4499-4323-96cc-23c78f693d2a None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:01.824276 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:01.841007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3118e327-4499-4323-96cc-23c78f693d2a None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:01.874082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68dc6b9-3be1-41e4-a1ac-b3d0f663a090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:01.874696 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68dc6b9-3be1-41e4-a1ac-b3d0f663a090 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:01.875007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68dc6b9-3be1-41e4-a1ac-b3d0f663a090 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:01.875279 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68dc6b9-3be1-41e4-a1ac-b3d0f663a090 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:01.885820 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68dc6b9-3be1-41e4-a1ac-b3d0f663a090 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:07:01.890520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-e68dc6b9-3be1-41e4-a1ac-b3d0f663a090 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:07:01.890520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:07:01.890520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:07:01.890520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:07:01.890520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:07:01.890520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:07:01.890520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:07:01.890520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:07:01.890520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:07:01.890520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:07:01.890520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:07:01.890520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:07:01.890520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:07:01.890520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:07:01.890520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:07:01.890520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:07:01.891649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:07:01.891649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:07:01.891649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:07:01.891649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:07:01.891649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:07:01.891649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:07:01.891649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:07:01.891649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:07:01.891649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:07:01.891649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:07:01.891649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:07:01.891649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:07:01.891649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:07:01.891649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:07:01.891649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:07:01.891649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:07:01.893068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:07:01.893068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:07:01.893068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:07:01.893068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:07:01.893068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:07:01.893068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:07:01.893068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:07:01.893068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:07:01.893068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:07:01.893068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:07:01.893068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:07:01.893068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:07:01.893068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 539/1089] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:07:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:07:01.894180 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edac26da-35cc-4f05-b9e1-b1701dc4352a None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:01.897875 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68dc6b9-3be1-41e4-a1ac-b3d0f663a090 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:01.948946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:01.994625 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edac26da-35cc-4f05-b9e1-b1701dc4352a None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:01.997679 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68dc6b9-3be1-41e4-a1ac-b3d0f663a090 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:02.028747 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee823e89-4b22-4b3d-910e-6b9ac1e1f5b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:02.029448 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee823e89-4b22-4b3d-910e-6b9ac1e1f5b3 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:02.029795 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee823e89-4b22-4b3d-910e-6b9ac1e1f5b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:02.030131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee823e89-4b22-4b3d-910e-6b9ac1e1f5b3 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:02.031225 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5955c6b9-428a-4a37-94ab-4e761273a53b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:02.031835 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5955c6b9-428a-4a37-94ab-4e761273a53b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:02.032117 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5955c6b9-428a-4a37-94ab-4e761273a53b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:02.032388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5955c6b9-428a-4a37-94ab-4e761273a53b None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:02.033042 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5955c6b9-428a-4a37-94ab-4e761273a53b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:02.033344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5955c6b9-428a-4a37-94ab-4e761273a53b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:02.034710 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5955c6b9-428a-4a37-94ab-4e761273a53b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:02.040520 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee823e89-4b22-4b3d-910e-6b9ac1e1f5b3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:07:02.042796 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 540/1090] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:07:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:07:02.047913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-ee823e89-4b22-4b3d-910e-6b9ac1e1f5b3 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 01:07:02.047913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:07:02.047913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:07:02.047913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:07:02.047913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:07:02.047913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:07:02.047913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:07:02.047913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:07:02.047913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:07:02.047913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:07:02.047913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:07:02.047913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:07:02.047913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:07:02.047913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:07:02.047913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:07:02.047913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:07:02.049004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:07:02.049004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:07:02.049004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:07:02.049004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:07:02.049004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:07:02.049004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:07:02.049004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:07:02.049004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:07:02.049004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:07:02.049004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:07:02.049004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:07:02.049004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:07:02.049004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:07:02.049004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:07:02.049004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:07:02.049004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:07:02.050046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:07:02.050046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:07:02.050046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:07:02.050046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:07:02.050046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:07:02.050046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:07:02.050046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:07:02.050046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:07:02.050046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:07:02.050046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:07:02.050046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 01:07:02.050046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:07:02.050046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 551/1091] 10.222.0.15 () {56 vars in 1120 bytes} [Tue Sep 13 01:07:01 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:07:02.053157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee823e89-4b22-4b3d-910e-6b9ac1e1f5b3 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:02.053274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5955c6b9-428a-4a37-94ab-4e761273a53b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:02.065308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:02.151274 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee823e89-4b22-4b3d-910e-6b9ac1e1f5b3 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:02.153335 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5955c6b9-428a-4a37-94ab-4e761273a53b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:02.165331 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:02.185169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b454aafa-34c2-4b90-858d-4770814740e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:02.185778 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b454aafa-34c2-4b90-858d-4770814740e3 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:02.186057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b454aafa-34c2-4b90-858d-4770814740e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:02.186331 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b454aafa-34c2-4b90-858d-4770814740e3 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:02.186564 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2973a852-690c-4907-bfbe-0f90806ff623 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:02.187024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b454aafa-34c2-4b90-858d-4770814740e3 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:02.187240 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2973a852-690c-4907-bfbe-0f90806ff623 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:02.187356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b454aafa-34c2-4b90-858d-4770814740e3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:02.187789 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2973a852-690c-4907-bfbe-0f90806ff623 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:02.188042 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2973a852-690c-4907-bfbe-0f90806ff623 None admin] PATH_INFO: `/v3/users/42fe3f2b90dc4d33b1f0e1e77da63224` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:02.188928 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b454aafa-34c2-4b90-858d-4770814740e3 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:02.189164 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2973a852-690c-4907-bfbe-0f90806ff623 None admin] RBAC: Authorizing `identity:get_user(user_id=42fe3f2b90dc4d33b1f0e1e77da63224)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:02.190565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2973a852-690c-4907-bfbe-0f90806ff623 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:02.192208 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 541/1092] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:07:02 2022] GET /identity/v3/users/42fe3f2b90dc4d33b1f0e1e77da63224 => generated 285 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:02.198884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 552/1093] 10.222.0.15 () {56 vars in 1115 bytes} [Tue Sep 13 01:07:02 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:02.199039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2973a852-690c-4907-bfbe-0f90806ff623 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:02.202293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b454aafa-34c2-4b90-858d-4770814740e3 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:02.272814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:02.295331 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2973a852-690c-4907-bfbe-0f90806ff623 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:02.302202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b454aafa-34c2-4b90-858d-4770814740e3 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:02.329053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e0fa87-8e46-4323-9e40-0c46852f546c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:02.329649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e0fa87-8e46-4323-9e40-0c46852f546c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:02.330028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e0fa87-8e46-4323-9e40-0c46852f546c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:02.330192 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e0fa87-8e46-4323-9e40-0c46852f546c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:02.335463 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb8b02c-fc8b-47db-9356-2020f454dc2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:02.336080 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb8b02c-fc8b-47db-9356-2020f454dc2a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:02.336357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb8b02c-fc8b-47db-9356-2020f454dc2a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:02.336626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb8b02c-fc8b-47db-9356-2020f454dc2a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:02.347123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb8b02c-fc8b-47db-9356-2020f454dc2a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:02.349066 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb8b02c-fc8b-47db-9356-2020f454dc2a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:02.362739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-cdb8b02c-fc8b-47db-9356-2020f454dc2a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:07:02.362739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:07:02.362739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:07:02.362739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:07:02.362739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:07:02.362739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:07:02.362739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:07:02.362739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:07:02.362739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:07:02.362739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:07:02.362739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:07:02.362739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:07:02.362739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:07:02.362739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:07:02.362739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:07:02.362739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:07:02.364002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:07:02.364002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:07:02.364002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:07:02.364002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:07:02.364002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:07:02.364002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:07:02.364002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:07:02.364002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:07:02.364002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:07:02.364002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:07:02.364002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:07:02.364002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:07:02.364002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:07:02.364002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:07:02.364002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:07:02.364002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:07:02.364002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:07:02.365104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:07:02.365104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:07:02.365104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:07:02.365104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 553/1094] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:07:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:07:02.369593 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb8b02c-fc8b-47db-9356-2020f454dc2a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:02.398207 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:02.399110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e0fa87-8e46-4323-9e40-0c46852f546c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:02.400615 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e0fa87-8e46-4323-9e40-0c46852f546c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:02.474966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:02.481545 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb8b02c-fc8b-47db-9356-2020f454dc2a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:02.498343 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 542/1095] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:07:02 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 01:07:02.504895 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e0fa87-8e46-4323-9e40-0c46852f546c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:02.515471 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a49bbe-c3e0-4048-a1c0-00891fb9766d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:02.516108 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a49bbe-c3e0-4048-a1c0-00891fb9766d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:02.516387 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a49bbe-c3e0-4048-a1c0-00891fb9766d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:02.516659 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a49bbe-c3e0-4048-a1c0-00891fb9766d None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:02.517294 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a49bbe-c3e0-4048-a1c0-00891fb9766d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:02.517597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a49bbe-c3e0-4048-a1c0-00891fb9766d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:02.518972 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a49bbe-c3e0-4048-a1c0-00891fb9766d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:02.531271 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 554/1096] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:07:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:07:02.536751 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a49bbe-c3e0-4048-a1c0-00891fb9766d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:02.600746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e0fa87-8e46-4323-9e40-0c46852f546c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:02.621596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:02.633875 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a49bbe-c3e0-4048-a1c0-00891fb9766d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:02.635529 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226e35ca-12c4-42fd-aa1b-fc5ca3135c48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:02.636180 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226e35ca-12c4-42fd-aa1b-fc5ca3135c48 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:02.636455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226e35ca-12c4-42fd-aa1b-fc5ca3135c48 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:02.636730 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226e35ca-12c4-42fd-aa1b-fc5ca3135c48 None admin] PATH_INFO: `/v3/projects/656038fa03254d489978b4958195a414` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:02.637821 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226e35ca-12c4-42fd-aa1b-fc5ca3135c48 None admin] RBAC: Authorizing `identity:get_project(project_id=656038fa03254d489978b4958195a414)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:02.639263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226e35ca-12c4-42fd-aa1b-fc5ca3135c48 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:02.640878 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 543/1097] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:07:02 2022] GET /identity/v3/projects/656038fa03254d489978b4958195a414 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:07:02.647395 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226e35ca-12c4-42fd-aa1b-fc5ca3135c48 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:02.672446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a813fd0-0e79-425f-b982-4b3e8a8f6016 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:02.672999 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a813fd0-0e79-425f-b982-4b3e8a8f6016 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:02.673264 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a813fd0-0e79-425f-b982-4b3e8a8f6016 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:02.673535 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a813fd0-0e79-425f-b982-4b3e8a8f6016 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:02.681052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a813fd0-0e79-425f-b982-4b3e8a8f6016 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:02.682451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a813fd0-0e79-425f-b982-4b3e8a8f6016 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:02.692497 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:02.694960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-1a813fd0-0e79-425f-b982-4b3e8a8f6016 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 01:07:02.694960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:07:02.694960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:07:02.694960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:07:02.694960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:07:02.694960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:07:02.694960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:07:02.694960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:07:02.694960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:07:02.694960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:07:02.694960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:07:02.694960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:07:02.694960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:07:02.694960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:07:02.694960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:07:02.694960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:07:02.695906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:07:02.695906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:07:02.695906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:07:02.695906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:07:02.695906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:07:02.695906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:07:02.695906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:07:02.695906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:07:02.695906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:07:02.695906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:07:02.695906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:07:02.695906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:07:02.695906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:07:02.695906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:07:02.695906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:07:02.695906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:07:02.695906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:07:02.695906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:07:02.696901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:07:02.696901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:07:02.696901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:07:02.696901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:07:02.696901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:07:02.696901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:07:02.696901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:07:02.696901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:07:02.696901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 01:07:02.696901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:07:02.696901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 555/1098] 10.222.0.15 () {56 vars in 1093 bytes} [Tue Sep 13 01:07:02 2022] GET /identity/v3/users/nova => generated 82 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:07:02.701698 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a813fd0-0e79-425f-b982-4b3e8a8f6016 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:02.749327 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226e35ca-12c4-42fd-aa1b-fc5ca3135c48 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:02.783277 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea2ad40-a895-49fa-990e-7b44f58dfeb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:02.783871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea2ad40-a895-49fa-990e-7b44f58dfeb9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:02.784146 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea2ad40-a895-49fa-990e-7b44f58dfeb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:02.784414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea2ad40-a895-49fa-990e-7b44f58dfeb9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:02.785121 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea2ad40-a895-49fa-990e-7b44f58dfeb9 None admin] RBAC: Adding query filter params (role.id=66a36c858aab40688080eed6ceace3a0, scope.project.id=656038fa03254d489978b4958195a414, user.id=42fe3f2b90dc4d33b1f0e1e77da63224) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:02.785980 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea2ad40-a895-49fa-990e-7b44f58dfeb9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:02.787420 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea2ad40-a895-49fa-990e-7b44f58dfeb9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:02.792962 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 544/1099] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:07:02 2022] GET /identity/v3/role_assignments?user.id=42fe3f2b90dc4d33b1f0e1e77da63224&scope.project.id=656038fa03254d489978b4958195a414&role.id=66a36c858aab40688080eed6ceace3a0 => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:07:02.807419 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a813fd0-0e79-425f-b982-4b3e8a8f6016 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:02.846067 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7d2e50-a5a9-45d4-8d3d-ed6464e59412 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:02.846702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7d2e50-a5a9-45d4-8d3d-ed6464e59412 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:02.847017 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7d2e50-a5a9-45d4-8d3d-ed6464e59412 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:02.847291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7d2e50-a5a9-45d4-8d3d-ed6464e59412 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:02.848585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7d2e50-a5a9-45d4-8d3d-ed6464e59412 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:02.848893 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7d2e50-a5a9-45d4-8d3d-ed6464e59412 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:02.850363 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7d2e50-a5a9-45d4-8d3d-ed6464e59412 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:02.869819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 556/1100] 10.222.0.15 () {56 vars in 1133 bytes} [Tue Sep 13 01:07:02 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:07:02.875741 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea2ad40-a895-49fa-990e-7b44f58dfeb9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:02.981439 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea2ad40-a895-49fa-990e-7b44f58dfeb9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:02.990360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:03.015082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d5b2ca-cc02-4746-bfc9-5935d36f7c2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:03.015695 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d5b2ca-cc02-4746-bfc9-5935d36f7c2b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:03.015975 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d5b2ca-cc02-4746-bfc9-5935d36f7c2b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:03.016248 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d5b2ca-cc02-4746-bfc9-5935d36f7c2b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:03.083131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d5b2ca-cc02-4746-bfc9-5935d36f7c2b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:03.084579 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d5b2ca-cc02-4746-bfc9-5935d36f7c2b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:03.182107 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 545/1101] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:07:02 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 13 01:07:03.188141 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7d2e50-a5a9-45d4-8d3d-ed6464e59412 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:03.196916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:03.284920 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7d2e50-a5a9-45d4-8d3d-ed6464e59412 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:03.318981 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd6c7ba-b180-4b0c-bd4e-d89135fa1cb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:03.319601 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd6c7ba-b180-4b0c-bd4e-d89135fa1cb3 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:03.319873 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd6c7ba-b180-4b0c-bd4e-d89135fa1cb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:03.320137 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd6c7ba-b180-4b0c-bd4e-d89135fa1cb3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:03.329817 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd6c7ba-b180-4b0c-bd4e-d89135fa1cb3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:03.331272 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd6c7ba-b180-4b0c-bd4e-d89135fa1cb3 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:03.345223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-dfd6c7ba-b180-4b0c-bd4e-d89135fa1cb3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:07:03.345223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:07:03.345223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:07:03.345223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:07:03.345223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:07:03.345223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:07:03.345223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:07:03.345223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:07:03.345223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:07:03.345223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:07:03.345223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:07:03.345223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:07:03.345223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:07:03.345223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:07:03.345223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:07:03.345223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:07:03.346346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:07:03.346346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:07:03.346346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:07:03.346346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:07:03.346346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:07:03.346346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:07:03.346346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:07:03.346346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:07:03.346346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:07:03.346346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:07:03.346346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:07:03.346346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:07:03.346346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:07:03.346346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:07:03.346346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:07:03.346346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:07:03.346346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:07:03.347477 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:07:03.347477 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:07:03.347477 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:07:03.351011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 557/1102] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:07:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:07:03.352506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d5b2ca-cc02-4746-bfc9-5935d36f7c2b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:03.362521 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:03.453280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d5b2ca-cc02-4746-bfc9-5935d36f7c2b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:03.487262 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64dd85ce-931b-4fb3-b5f1-c6b203661f1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:03.487863 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dd85ce-931b-4fb3-b5f1-c6b203661f1b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:03.488136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dd85ce-931b-4fb3-b5f1-c6b203661f1b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:03.489263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dd85ce-931b-4fb3-b5f1-c6b203661f1b None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:03.489914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dd85ce-931b-4fb3-b5f1-c6b203661f1b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:03.490217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dd85ce-931b-4fb3-b5f1-c6b203661f1b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:03.491598 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dd85ce-931b-4fb3-b5f1-c6b203661f1b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:03.505927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 546/1103] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:07:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:03.511537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd6c7ba-b180-4b0c-bd4e-d89135fa1cb3 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:03.570025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:03.610881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd6c7ba-b180-4b0c-bd4e-d89135fa1cb3 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:03.643516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc46a3a-71a1-4876-bfef-7bbe1efd7d7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:03.644127 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc46a3a-71a1-4876-bfef-7bbe1efd7d7a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:03.644400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc46a3a-71a1-4876-bfef-7bbe1efd7d7a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:03.644692 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc46a3a-71a1-4876-bfef-7bbe1efd7d7a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:03.651596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc46a3a-71a1-4876-bfef-7bbe1efd7d7a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:03.653239 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc46a3a-71a1-4876-bfef-7bbe1efd7d7a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:03.664073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-bcc46a3a-71a1-4876-bfef-7bbe1efd7d7a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:07:03.664073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:07:03.664073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:07:03.664073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:07:03.664073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:07:03.664073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:07:03.664073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:07:03.664073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:07:03.664073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:07:03.664073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:07:03.664073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:07:03.664073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:07:03.664073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:07:03.664073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:07:03.664073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:07:03.665149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:07:03.665149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:07:03.665149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:07:03.665149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:07:03.665149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:07:03.665149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:07:03.665149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:07:03.665149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:07:03.665149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:07:03.665149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:07:03.665149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:07:03.665149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:07:03.665149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:07:03.665149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:07:03.665149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:07:03.665149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:07:03.665149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:07:03.666202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:07:03.666202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:07:03.666202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:07:03.666202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:07:03.666202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:07:03.666202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:07:03.666202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:07:03.666202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:07:03.667014 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 558/1104] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:07:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:07:03.671006 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64dd85ce-931b-4fb3-b5f1-c6b203661f1b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:03.737987 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:03.771284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64dd85ce-931b-4fb3-b5f1-c6b203661f1b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:03.803272 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e7e710-196b-46b2-81ed-78897d415308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:03.803862 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e7e710-196b-46b2-81ed-78897d415308 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:03.804137 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e7e710-196b-46b2-81ed-78897d415308 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:03.804405 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e7e710-196b-46b2-81ed-78897d415308 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:03.805074 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e7e710-196b-46b2-81ed-78897d415308 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:03.805376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e7e710-196b-46b2-81ed-78897d415308 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:03.806780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e7e710-196b-46b2-81ed-78897d415308 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:03.819073 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 547/1105] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:07:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:03.825108 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc46a3a-71a1-4876-bfef-7bbe1efd7d7a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:03.921918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc46a3a-71a1-4876-bfef-7bbe1efd7d7a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:03.941566 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:03.958708 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b796e36-a408-4ff4-a5b3-b235071ee28b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:03.959329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b796e36-a408-4ff4-a5b3-b235071ee28b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:03.959603 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b796e36-a408-4ff4-a5b3-b235071ee28b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:03.959869 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b796e36-a408-4ff4-a5b3-b235071ee28b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:03.960575 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b796e36-a408-4ff4-a5b3-b235071ee28b None admin] RBAC: Adding query filter params (role.id=11f2e919cae541428f8c564b7b67eea0, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=2f2da4d2d3604d16b58dc25b8c17c49a) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:03.960877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b796e36-a408-4ff4-a5b3-b235071ee28b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:03.962270 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b796e36-a408-4ff4-a5b3-b235071ee28b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:03.967916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 559/1106] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:07:03 2022] GET /identity/v3/role_assignments?user.id=2f2da4d2d3604d16b58dc25b8c17c49a&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=11f2e919cae541428f8c564b7b67eea0 => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:04.976259 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c36e14-5ef9-40a0-93df-7e20bd2415cf None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:04.976897 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c36e14-5ef9-40a0-93df-7e20bd2415cf None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:04.976897 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c36e14-5ef9-40a0-93df-7e20bd2415cf None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:04.978125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 548/1107] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:07:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:07:04.983234 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f4ea66-837e-4cf6-b027-d02fdf135cb5 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:04.983770 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f4ea66-837e-4cf6-b027-d02fdf135cb5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:04.983770 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f4ea66-837e-4cf6-b027-d02fdf135cb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:05.001937 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-65f4ea66-837e-4cf6-b027-d02fdf135cb5 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:07:05.003126 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-65f4ea66-837e-4cf6-b027-d02fdf135cb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:07:05.066292 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 560/1108] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:07:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 01:07:05.333164 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c36e14-5ef9-40a0-93df-7e20bd2415cf None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:05.431617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c36e14-5ef9-40a0-93df-7e20bd2415cf None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:05.456855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:05.464726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5a663e-f46e-4eef-8326-d31a0e71f887 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:05.465346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5a663e-f46e-4eef-8326-d31a0e71f887 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:05.465619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5a663e-f46e-4eef-8326-d31a0e71f887 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:05.465884 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5a663e-f46e-4eef-8326-d31a0e71f887 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:05.466996 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5a663e-f46e-4eef-8326-d31a0e71f887 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:05.468402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5a663e-f46e-4eef-8326-d31a0e71f887 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:05.469941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 549/1109] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:07:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:05.476166 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f4ea66-837e-4cf6-b027-d02fdf135cb5 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:05.573805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f4ea66-837e-4cf6-b027-d02fdf135cb5 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:05.586962 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:05.607399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01934bfd-d262-49cb-a0b6-481866650431 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:05.607995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01934bfd-d262-49cb-a0b6-481866650431 None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:05.608274 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01934bfd-d262-49cb-a0b6-481866650431 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:05.608546 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01934bfd-d262-49cb-a0b6-481866650431 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:05.609187 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01934bfd-d262-49cb-a0b6-481866650431 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:05.610590 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01934bfd-d262-49cb-a0b6-481866650431 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:05.632186 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 561/1110] 10.222.0.15 () {60 vars in 1132 bytes} [Tue Sep 13 01:07:05 2022] POST /identity/v3/users => generated 286 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:07:07.803024 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e223d2c1-020a-4942-a654-4d4483e606eb None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:07.803617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e223d2c1-020a-4942-a654-4d4483e606eb None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:07.803617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e223d2c1-020a-4942-a654-4d4483e606eb None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:07.804892 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 550/1111] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:07:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:07:07.810176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d2e495-4ad7-4783-b5c5-63243b26e990 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:07.810619 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d2e495-4ad7-4783-b5c5-63243b26e990 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:07.810716 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d2e495-4ad7-4783-b5c5-63243b26e990 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:07.828606 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-03d2e495-4ad7-4783-b5c5-63243b26e990 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:07:07.829784 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-03d2e495-4ad7-4783-b5c5-63243b26e990 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:07:07.892959 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 562/1112] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:07:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 01:07:08.169011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e223d2c1-020a-4942-a654-4d4483e606eb None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:08.266469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e223d2c1-020a-4942-a654-4d4483e606eb None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:08.298635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6f04d5-7879-4aaa-998d-a33ec45ac488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:08.299270 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6f04d5-7879-4aaa-998d-a33ec45ac488 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:08.299547 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6f04d5-7879-4aaa-998d-a33ec45ac488 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:08.299815 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6f04d5-7879-4aaa-998d-a33ec45ac488 None admin] PATH_INFO: `/v3/roles/cd616e816a394068867091116a3f1ce0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:08.301696 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6f04d5-7879-4aaa-998d-a33ec45ac488 None admin] RBAC: Authorizing `identity:get_role(role_id=cd616e816a394068867091116a3f1ce0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:08.303062 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6f04d5-7879-4aaa-998d-a33ec45ac488 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:08.304340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 551/1113] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:07:08 2022] GET /identity/v3/roles/cd616e816a394068867091116a3f1ce0 => generated 223 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:08.309964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d2e495-4ad7-4783-b5c5-63243b26e990 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:08.403045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:08.407039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d2e495-4ad7-4783-b5c5-63243b26e990 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:08.442075 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a17b69-ffd4-4c30-8579-6bac594a8579 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:08.442636 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a17b69-ffd4-4c30-8579-6bac594a8579 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:08.442937 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a17b69-ffd4-4c30-8579-6bac594a8579 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:08.443206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a17b69-ffd4-4c30-8579-6bac594a8579 None admin] PATH_INFO: `/v3/users/68752dbe18934eb4a1e03d3924ec5c6e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:08.456307 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a17b69-ffd4-4c30-8579-6bac594a8579 None admin] RBAC: Authorizing `identity:get_user(user_id=68752dbe18934eb4a1e03d3924ec5c6e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:08.457945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a17b69-ffd4-4c30-8579-6bac594a8579 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:08.459534 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 563/1114] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:07:08 2022] GET /identity/v3/users/68752dbe18934eb4a1e03d3924ec5c6e => generated 286 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:08.465251 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6f04d5-7879-4aaa-998d-a33ec45ac488 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:08.525782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:08.562808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6f04d5-7879-4aaa-998d-a33ec45ac488 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:08.595682 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260340a2-6f29-45dc-830d-aab42c8eb74f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:08.596273 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260340a2-6f29-45dc-830d-aab42c8eb74f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:08.596548 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260340a2-6f29-45dc-830d-aab42c8eb74f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:08.596813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260340a2-6f29-45dc-830d-aab42c8eb74f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:08.660614 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260340a2-6f29-45dc-830d-aab42c8eb74f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:08.662019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260340a2-6f29-45dc-830d-aab42c8eb74f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:08.718702 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:08.755129 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 552/1115] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:07:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 01:07:08.760996 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a17b69-ffd4-4c30-8579-6bac594a8579 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:08.855996 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a17b69-ffd4-4c30-8579-6bac594a8579 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:08.888193 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b20e7e-6006-40f4-b60b-01395d7e1be6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:08.888803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b20e7e-6006-40f4-b60b-01395d7e1be6 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:08.889073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b20e7e-6006-40f4-b60b-01395d7e1be6 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:08.889339 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b20e7e-6006-40f4-b60b-01395d7e1be6 None admin] PATH_INFO: `/v3/projects/656038fa03254d489978b4958195a414` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:08.890413 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b20e7e-6006-40f4-b60b-01395d7e1be6 None admin] RBAC: Authorizing `identity:get_project(project_id=656038fa03254d489978b4958195a414)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:08.891834 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b20e7e-6006-40f4-b60b-01395d7e1be6 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:08.893411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 564/1116] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:07:08 2022] GET /identity/v3/projects/656038fa03254d489978b4958195a414 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:08.899931 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260340a2-6f29-45dc-830d-aab42c8eb74f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:08.994875 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260340a2-6f29-45dc-830d-aab42c8eb74f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:09.027383 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff5dc6b-b688-44e8-8b2d-598b3f10306f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:09.027972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff5dc6b-b688-44e8-8b2d-598b3f10306f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:09.028246 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff5dc6b-b688-44e8-8b2d-598b3f10306f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:09.028512 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff5dc6b-b688-44e8-8b2d-598b3f10306f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:09.029219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff5dc6b-b688-44e8-8b2d-598b3f10306f None admin] RBAC: Adding query filter params (role.id=cd616e816a394068867091116a3f1ce0, scope.project.id=656038fa03254d489978b4958195a414, user.id=68752dbe18934eb4a1e03d3924ec5c6e) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:09.029520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff5dc6b-b688-44e8-8b2d-598b3f10306f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:09.030997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff5dc6b-b688-44e8-8b2d-598b3f10306f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:09.036769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 553/1117] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:07:08 2022] GET /identity/v3/role_assignments?user.id=68752dbe18934eb4a1e03d3924ec5c6e&scope.project.id=656038fa03254d489978b4958195a414&role.id=cd616e816a394068867091116a3f1ce0 => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:11.161030 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6129bda3-eb4e-4129-9247-e8952ade30c1 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:11.161586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6129bda3-eb4e-4129-9247-e8952ade30c1 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:11.161586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6129bda3-eb4e-4129-9247-e8952ade30c1 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:11.162934 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 565/1118] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:07:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:07:11.168202 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f34d00-d450-4851-9d24-9e1ca54cf713 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:11.168569 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f34d00-d450-4851-9d24-9e1ca54cf713 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:11.168784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f34d00-d450-4851-9d24-9e1ca54cf713 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:11.186774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b1f34d00-d450-4851-9d24-9e1ca54cf713 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:07:11.188227 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b1f34d00-d450-4851-9d24-9e1ca54cf713 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:07:11.210575 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:11.252657 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 554/1119] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:07:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:07:11.523212 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6129bda3-eb4e-4129-9247-e8952ade30c1 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:11.550447 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:11.625023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6129bda3-eb4e-4129-9247-e8952ade30c1 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:11.656627 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2c891d-548e-4f63-85b4-389e5d515983 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:11.657236 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2c891d-548e-4f63-85b4-389e5d515983 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:11.657510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2c891d-548e-4f63-85b4-389e5d515983 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:11.657784 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2c891d-548e-4f63-85b4-389e5d515983 None admin] PATH_INFO: `/v3/roles/cd616e816a394068867091116a3f1ce0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:11.659163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2c891d-548e-4f63-85b4-389e5d515983 None admin] RBAC: Authorizing `identity:get_role(role_id=cd616e816a394068867091116a3f1ce0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:11.661023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2c891d-548e-4f63-85b4-389e5d515983 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:11.662315 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 566/1120] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:07:11 2022] GET /identity/v3/roles/cd616e816a394068867091116a3f1ce0 => generated 223 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:11.668147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f34d00-d450-4851-9d24-9e1ca54cf713 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:11.763572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f34d00-d450-4851-9d24-9e1ca54cf713 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:11.796133 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d532ee33-64a9-4e8c-8778-214566b45bd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:11.796690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d532ee33-64a9-4e8c-8778-214566b45bd9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:11.796973 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d532ee33-64a9-4e8c-8778-214566b45bd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:11.797245 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d532ee33-64a9-4e8c-8778-214566b45bd9 None admin] PATH_INFO: `/v3/users/68752dbe18934eb4a1e03d3924ec5c6e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:11.798347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d532ee33-64a9-4e8c-8778-214566b45bd9 None admin] RBAC: Authorizing `identity:get_user(user_id=68752dbe18934eb4a1e03d3924ec5c6e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:11.799788 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d532ee33-64a9-4e8c-8778-214566b45bd9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:11.801838 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 555/1121] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:07:11 2022] GET /identity/v3/users/68752dbe18934eb4a1e03d3924ec5c6e => generated 286 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:11.807282 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2c891d-548e-4f63-85b4-389e5d515983 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:11.893912 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:11.902270 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2c891d-548e-4f63-85b4-389e5d515983 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:11.934451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d299d9-9d26-45c2-a22e-416c8dc7735e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:11.935083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d299d9-9d26-45c2-a22e-416c8dc7735e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:11.935355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d299d9-9d26-45c2-a22e-416c8dc7735e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:11.935620 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d299d9-9d26-45c2-a22e-416c8dc7735e None admin] PATH_INFO: `/v3/projects/656038fa03254d489978b4958195a414` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:11.936695 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d299d9-9d26-45c2-a22e-416c8dc7735e None admin] RBAC: Authorizing `identity:get_project(project_id=656038fa03254d489978b4958195a414)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:11.938351 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d299d9-9d26-45c2-a22e-416c8dc7735e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:11.939960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 567/1122] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:07:11 2022] GET /identity/v3/projects/656038fa03254d489978b4958195a414 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:11.946286 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d532ee33-64a9-4e8c-8778-214566b45bd9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:11.972347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:12.046064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d532ee33-64a9-4e8c-8778-214566b45bd9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:12.079384 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00278847-f289-41bd-8a39-1ac864085d71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:12.079965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00278847-f289-41bd-8a39-1ac864085d71 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:12.080240 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00278847-f289-41bd-8a39-1ac864085d71 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:12.080509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00278847-f289-41bd-8a39-1ac864085d71 None admin] PATH_INFO: `/v3/projects/656038fa03254d489978b4958195a414/users/68752dbe18934eb4a1e03d3924ec5c6e/roles/cd616e816a394068867091116a3f1ce0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:12.083205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00278847-f289-41bd-8a39-1ac864085d71 None admin] RBAC: Authorizing `identity:create_grant(project_id=656038fa03254d489978b4958195a414, user_id=68752dbe18934eb4a1e03d3924ec5c6e, role_id=cd616e816a394068867091116a3f1ce0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:12.084628 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00278847-f289-41bd-8a39-1ac864085d71 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:12.096691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 556/1123] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:07:11 2022] PUT /identity/v3/projects/656038fa03254d489978b4958195a414/users/68752dbe18934eb4a1e03d3924ec5c6e/roles/cd616e816a394068867091116a3f1ce0 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:07:14.176575 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10b72d3-8482-4ce2-9297-5b6b604f64f2 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:14.177107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10b72d3-8482-4ce2-9297-5b6b604f64f2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:14.177107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10b72d3-8482-4ce2-9297-5b6b604f64f2 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:14.178438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 568/1124] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:07:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:07:14.183838 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3319d907-c865-476a-aa24-5c7b399b5d68 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:14.184257 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3319d907-c865-476a-aa24-5c7b399b5d68 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:14.184368 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3319d907-c865-476a-aa24-5c7b399b5d68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:14.202252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-3319d907-c865-476a-aa24-5c7b399b5d68 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:07:14.203440 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3319d907-c865-476a-aa24-5c7b399b5d68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:07:14.266971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 557/1125] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:07:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:07:14.529484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10b72d3-8482-4ce2-9297-5b6b604f64f2 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:14.626706 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10b72d3-8482-4ce2-9297-5b6b604f64f2 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:14.660555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54c341d-5426-440f-ade0-d316e2597cae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:14.661164 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54c341d-5426-440f-ade0-d316e2597cae None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:14.661438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54c341d-5426-440f-ade0-d316e2597cae None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:14.661705 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54c341d-5426-440f-ade0-d316e2597cae None admin] PATH_INFO: `/v3/roles/cd616e816a394068867091116a3f1ce0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:14.663074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54c341d-5426-440f-ade0-d316e2597cae None admin] RBAC: Authorizing `identity:get_role(role_id=cd616e816a394068867091116a3f1ce0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:14.664417 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54c341d-5426-440f-ade0-d316e2597cae None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:14.665708 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 569/1126] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:07:14 2022] GET /identity/v3/roles/cd616e816a394068867091116a3f1ce0 => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:14.671426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3319d907-c865-476a-aa24-5c7b399b5d68 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:14.689003 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:14.768326 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3319d907-c865-476a-aa24-5c7b399b5d68 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:14.800907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4faf32-2ead-49fe-acf5-52bdd5d2c14b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:14.801466 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4faf32-2ead-49fe-acf5-52bdd5d2c14b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:14.801745 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4faf32-2ead-49fe-acf5-52bdd5d2c14b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:14.802014 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4faf32-2ead-49fe-acf5-52bdd5d2c14b None admin] PATH_INFO: `/v3/users/68752dbe18934eb4a1e03d3924ec5c6e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:14.803137 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4faf32-2ead-49fe-acf5-52bdd5d2c14b None admin] RBAC: Authorizing `identity:get_user(user_id=68752dbe18934eb4a1e03d3924ec5c6e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:14.804556 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4faf32-2ead-49fe-acf5-52bdd5d2c14b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:14.806102 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 558/1127] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:07:14 2022] GET /identity/v3/users/68752dbe18934eb4a1e03d3924ec5c6e => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:14.811661 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54c341d-5426-440f-ade0-d316e2597cae None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:14.839431 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:14.908243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54c341d-5426-440f-ade0-d316e2597cae None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:14.940823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49bc308-a226-4b33-ba1d-61a391c79aed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:14.941949 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49bc308-a226-4b33-ba1d-61a391c79aed None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:14.942225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49bc308-a226-4b33-ba1d-61a391c79aed None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:14.942491 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49bc308-a226-4b33-ba1d-61a391c79aed None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:15.006236 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49bc308-a226-4b33-ba1d-61a391c79aed None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:15.007677 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49bc308-a226-4b33-ba1d-61a391c79aed None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:15.033321 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:15.102481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 570/1128] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:07:14 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 01:07:15.108346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4faf32-2ead-49fe-acf5-52bdd5d2c14b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:15.192204 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:15.207557 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4faf32-2ead-49fe-acf5-52bdd5d2c14b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:15.240565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c93035-fcfa-47aa-856d-7863dfb7e6d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:15.241183 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c93035-fcfa-47aa-856d-7863dfb7e6d8 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:15.241460 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c93035-fcfa-47aa-856d-7863dfb7e6d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:15.241730 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c93035-fcfa-47aa-856d-7863dfb7e6d8 None admin] PATH_INFO: `/v3/projects/656038fa03254d489978b4958195a414` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:15.242824 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c93035-fcfa-47aa-856d-7863dfb7e6d8 None admin] RBAC: Authorizing `identity:get_project(project_id=656038fa03254d489978b4958195a414)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:15.244274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c93035-fcfa-47aa-856d-7863dfb7e6d8 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:15.245875 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 559/1129] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:07:15 2022] GET /identity/v3/projects/656038fa03254d489978b4958195a414 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:15.251878 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49bc308-a226-4b33-ba1d-61a391c79aed None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:15.345688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49bc308-a226-4b33-ba1d-61a391c79aed None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:15.371580 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:15.379383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919834e9-91a9-449f-ab6e-cd0f97a18fb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:15.379968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919834e9-91a9-449f-ab6e-cd0f97a18fb7 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:15.380239 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919834e9-91a9-449f-ab6e-cd0f97a18fb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:15.380506 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919834e9-91a9-449f-ab6e-cd0f97a18fb7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:15.381207 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919834e9-91a9-449f-ab6e-cd0f97a18fb7 None admin] RBAC: Adding query filter params (role.id=cd616e816a394068867091116a3f1ce0, scope.project.id=656038fa03254d489978b4958195a414, user.id=68752dbe18934eb4a1e03d3924ec5c6e) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:15.381508 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919834e9-91a9-449f-ab6e-cd0f97a18fb7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:15.382926 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919834e9-91a9-449f-ab6e-cd0f97a18fb7 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:15.388429 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 571/1130] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:07:15 2022] GET /identity/v3/role_assignments?user.id=68752dbe18934eb4a1e03d3924ec5c6e&scope.project.id=656038fa03254d489978b4958195a414&role.id=cd616e816a394068867091116a3f1ce0 => generated 608 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:17.510683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e2f3af-2694-452c-8f5f-a31c7335b73c None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:17.511228 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e2f3af-2694-452c-8f5f-a31c7335b73c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:17.511332 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e2f3af-2694-452c-8f5f-a31c7335b73c None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:17.512597 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 560/1131] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:07:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:07:17.517792 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a98bec-80cf-4d95-85f3-7952ed3dbed7 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:17.518179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a98bec-80cf-4d95-85f3-7952ed3dbed7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:17.518330 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a98bec-80cf-4d95-85f3-7952ed3dbed7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:17.536427 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-81a98bec-80cf-4d95-85f3-7952ed3dbed7 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:07:17.537575 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-81a98bec-80cf-4d95-85f3-7952ed3dbed7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:07:17.603523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 572/1132] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:07:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:07:17.869548 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e2f3af-2694-452c-8f5f-a31c7335b73c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:17.967527 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e2f3af-2694-452c-8f5f-a31c7335b73c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:17.999957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b94a9a-4432-40c0-ba1f-afeb39befb49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:18.000568 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b94a9a-4432-40c0-ba1f-afeb39befb49 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:18.000845 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b94a9a-4432-40c0-ba1f-afeb39befb49 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:18.001114 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b94a9a-4432-40c0-ba1f-afeb39befb49 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:18.002205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b94a9a-4432-40c0-ba1f-afeb39befb49 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:18.004235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b94a9a-4432-40c0-ba1f-afeb39befb49 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:18.005753 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 561/1133] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:07:17 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:18.011490 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a98bec-80cf-4d95-85f3-7952ed3dbed7 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:18.105796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a98bec-80cf-4d95-85f3-7952ed3dbed7 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:18.127576 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:18.139027 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e01329a1-8c64-4f8f-998b-996bcdd8a61d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:18.139629 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01329a1-8c64-4f8f-998b-996bcdd8a61d None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:18.139902 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01329a1-8c64-4f8f-998b-996bcdd8a61d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:18.140169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01329a1-8c64-4f8f-998b-996bcdd8a61d None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:18.140808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e01329a1-8c64-4f8f-998b-996bcdd8a61d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:18.142210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e01329a1-8c64-4f8f-998b-996bcdd8a61d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:18.170114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 573/1134] 10.222.0.15 () {60 vars in 1141 bytes} [Tue Sep 13 01:07:18 2022] POST /identity/v3/projects => generated 311 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:07:20.312091 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867a2ae3-627e-4128-a028-694aab0608b0 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:20.312966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867a2ae3-627e-4128-a028-694aab0608b0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:20.312966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867a2ae3-627e-4128-a028-694aab0608b0 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:20.314211 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 562/1135] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:07:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:07:20.319479 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e3662f-6b85-4cf0-b894-0095a37122a9 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:20.319933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e3662f-6b85-4cf0-b894-0095a37122a9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:20.320031 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e3662f-6b85-4cf0-b894-0095a37122a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:20.337640 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-f9e3662f-6b85-4cf0-b894-0095a37122a9 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:07:20.338795 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f9e3662f-6b85-4cf0-b894-0095a37122a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:07:20.402238 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 574/1136] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:07:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 01:07:20.671944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867a2ae3-627e-4128-a028-694aab0608b0 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:20.693907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:20.771219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867a2ae3-627e-4128-a028-694aab0608b0 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:20.803550 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ef619a-448f-46b6-a57f-f03755c832d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:20.804160 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ef619a-448f-46b6-a57f-f03755c832d2 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:20.804435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ef619a-448f-46b6-a57f-f03755c832d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:20.804699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ef619a-448f-46b6-a57f-f03755c832d2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:20.805781 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ef619a-448f-46b6-a57f-f03755c832d2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:20.807214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ef619a-448f-46b6-a57f-f03755c832d2 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:20.808742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 563/1137] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:07:20 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:20.814667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e3662f-6b85-4cf0-b894-0095a37122a9 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:20.908618 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e3662f-6b85-4cf0-b894-0095a37122a9 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:20.911458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:20.942981 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fccb19bc-7f96-493b-aae1-b0fec1c2a2ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:20.943513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccb19bc-7f96-493b-aae1-b0fec1c2a2ba None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:20.943790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccb19bc-7f96-493b-aae1-b0fec1c2a2ba None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:20.944059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccb19bc-7f96-493b-aae1-b0fec1c2a2ba None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:20.944713 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fccb19bc-7f96-493b-aae1-b0fec1c2a2ba None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:20.946121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fccb19bc-7f96-493b-aae1-b0fec1c2a2ba None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:20.966313 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 575/1138] 10.222.0.15 () {60 vars in 1132 bytes} [Tue Sep 13 01:07:20 2022] POST /identity/v3/users => generated 286 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:07:23.095433 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c4abe4-84bd-4c25-9ae8-17f1a75a3e7c None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:23.095989 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c4abe4-84bd-4c25-9ae8-17f1a75a3e7c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:23.095989 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c4abe4-84bd-4c25-9ae8-17f1a75a3e7c None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:23.097888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 564/1139] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:07:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:07:23.103141 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24944aac-0c42-4705-bf0a-95ccf42cdf99 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:23.103549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24944aac-0c42-4705-bf0a-95ccf42cdf99 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:23.103695 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24944aac-0c42-4705-bf0a-95ccf42cdf99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:23.122090 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-24944aac-0c42-4705-bf0a-95ccf42cdf99 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:07:23.123300 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-24944aac-0c42-4705-bf0a-95ccf42cdf99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:07:23.192699 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 576/1140] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:07:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:07:23.459718 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c4abe4-84bd-4c25-9ae8-17f1a75a3e7c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:23.549647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:23.563461 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c4abe4-84bd-4c25-9ae8-17f1a75a3e7c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:23.599802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9dd627-0f69-4b81-8371-f8f1cddc71f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:23.600421 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9dd627-0f69-4b81-8371-f8f1cddc71f9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:23.600697 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9dd627-0f69-4b81-8371-f8f1cddc71f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:23.600965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9dd627-0f69-4b81-8371-f8f1cddc71f9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:23.610650 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9dd627-0f69-4b81-8371-f8f1cddc71f9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:07:23.616026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-9b9dd627-0f69-4b81-8371-f8f1cddc71f9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:07:23.616026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:07:23.616026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:07:23.616026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:07:23.616026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:07:23.616026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:07:23.616026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:07:23.616026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:07:23.616026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:07:23.616026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:07:23.616026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:07:23.616026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:07:23.616026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:07:23.616026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:07:23.616026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:07:23.616026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:07:23.617168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:07:23.617168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:07:23.617168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:07:23.617168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:07:23.617168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:07:23.617168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:07:23.617168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:07:23.617168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:07:23.617168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:07:23.617168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:07:23.617168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:07:23.617168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:07:23.617168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:07:23.617168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:07:23.617168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:07:23.617168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:07:23.618090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:07:23.618090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:07:23.618090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:07:23.618090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:07:23.618090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:07:23.618090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:07:23.618090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:07:23.618090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:07:23.618090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:07:23.618090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:07:23.618090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:07:23.618090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:07:23.618090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 565/1141] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:07:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:07:23.623208 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24944aac-0c42-4705-bf0a-95ccf42cdf99 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:23.709463 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:23.720719 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24944aac-0c42-4705-bf0a-95ccf42cdf99 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:23.754117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed9ec8f-9ac1-4d46-9d82-bad38d047442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:23.754709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed9ec8f-9ac1-4d46-9d82-bad38d047442 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:23.755017 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed9ec8f-9ac1-4d46-9d82-bad38d047442 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:23.755288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed9ec8f-9ac1-4d46-9d82-bad38d047442 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:23.755933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed9ec8f-9ac1-4d46-9d82-bad38d047442 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:23.756233 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed9ec8f-9ac1-4d46-9d82-bad38d047442 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:23.757579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed9ec8f-9ac1-4d46-9d82-bad38d047442 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:23.765808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 577/1142] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:07:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:23.772605 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9dd627-0f69-4b81-8371-f8f1cddc71f9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:23.870297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9dd627-0f69-4b81-8371-f8f1cddc71f9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:23.903208 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e6045a-514d-47d5-ba5e-cacd92e972cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:23.903824 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e6045a-514d-47d5-ba5e-cacd92e972cc None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:23.904103 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e6045a-514d-47d5-ba5e-cacd92e972cc None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:23.904375 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e6045a-514d-47d5-ba5e-cacd92e972cc None admin] PATH_INFO: `/v3/users/fffd672596964a97bfe3c947d0f5072a` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:23.917057 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e6045a-514d-47d5-ba5e-cacd92e972cc None admin] RBAC: Authorizing `identity:get_user(user_id=fffd672596964a97bfe3c947d0f5072a)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:23.918473 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e6045a-514d-47d5-ba5e-cacd92e972cc None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:23.920096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 566/1143] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:07:23 2022] GET /identity/v3/users/fffd672596964a97bfe3c947d0f5072a => generated 286 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:23.926356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed9ec8f-9ac1-4d46-9d82-bad38d047442 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:24.022777 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed9ec8f-9ac1-4d46-9d82-bad38d047442 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:24.055165 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6df4cc-45ce-42dd-89c1-29f788060a2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:24.055762 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6df4cc-45ce-42dd-89c1-29f788060a2b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:24.056043 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6df4cc-45ce-42dd-89c1-29f788060a2b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:24.056316 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6df4cc-45ce-42dd-89c1-29f788060a2b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:24.077246 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:24.122333 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6df4cc-45ce-42dd-89c1-29f788060a2b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:24.123783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6df4cc-45ce-42dd-89c1-29f788060a2b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:24.217524 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 578/1144] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:07:23 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 01:07:24.223886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e6045a-514d-47d5-ba5e-cacd92e972cc None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:24.232502 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:24.320179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e6045a-514d-47d5-ba5e-cacd92e972cc None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:24.352705 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810bfaaa-7705-4639-93e7-6e1491f8a109 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:24.353317 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810bfaaa-7705-4639-93e7-6e1491f8a109 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:24.353595 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810bfaaa-7705-4639-93e7-6e1491f8a109 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:24.353866 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810bfaaa-7705-4639-93e7-6e1491f8a109 None admin] PATH_INFO: `/v3/projects/b86e433947634f47a189f296c7fe86ad` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:24.354989 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810bfaaa-7705-4639-93e7-6e1491f8a109 None admin] RBAC: Authorizing `identity:get_project(project_id=b86e433947634f47a189f296c7fe86ad)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:24.356397 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810bfaaa-7705-4639-93e7-6e1491f8a109 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:24.357977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 567/1145] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:07:24 2022] GET /identity/v3/projects/b86e433947634f47a189f296c7fe86ad => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:24.363919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6df4cc-45ce-42dd-89c1-29f788060a2b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:24.414732 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:24.460911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6df4cc-45ce-42dd-89c1-29f788060a2b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:24.493228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab707b1f-9583-45e4-9089-ecc53f32f557 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:24.493817 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab707b1f-9583-45e4-9089-ecc53f32f557 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:24.494088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab707b1f-9583-45e4-9089-ecc53f32f557 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:24.494357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab707b1f-9583-45e4-9089-ecc53f32f557 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:24.495085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab707b1f-9583-45e4-9089-ecc53f32f557 None admin] RBAC: Adding query filter params (role.id=66a36c858aab40688080eed6ceace3a0, scope.project.id=b86e433947634f47a189f296c7fe86ad, user.id=fffd672596964a97bfe3c947d0f5072a) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:24.495388 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab707b1f-9583-45e4-9089-ecc53f32f557 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:24.496787 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab707b1f-9583-45e4-9089-ecc53f32f557 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:24.501921 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 579/1146] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:07:24 2022] GET /identity/v3/role_assignments?user.id=fffd672596964a97bfe3c947d0f5072a&scope.project.id=b86e433947634f47a189f296c7fe86ad&role.id=66a36c858aab40688080eed6ceace3a0 => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:07:26.589430 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c20776-1787-42d1-ae08-54cd253714b4 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:26.589952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c20776-1787-42d1-ae08-54cd253714b4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:26.589952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c20776-1787-42d1-ae08-54cd253714b4 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:26.591362 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 568/1147] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:07:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:07:26.596372 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bfc1fc-b2d6-48e5-8a6e-213bef55cd1a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:26.596770 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bfc1fc-b2d6-48e5-8a6e-213bef55cd1a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:26.596938 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bfc1fc-b2d6-48e5-8a6e-213bef55cd1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:26.614495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-47bfc1fc-b2d6-48e5-8a6e-213bef55cd1a None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:07:26.615674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-47bfc1fc-b2d6-48e5-8a6e-213bef55cd1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:07:26.680306 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 580/1148] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:07:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:07:26.946714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c20776-1787-42d1-ae08-54cd253714b4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:27.020053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:27.045376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c20776-1787-42d1-ae08-54cd253714b4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:27.078462 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f46ac99-0f51-4e26-9214-0f03444481c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:27.079103 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f46ac99-0f51-4e26-9214-0f03444481c0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:27.079402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f46ac99-0f51-4e26-9214-0f03444481c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:27.079713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f46ac99-0f51-4e26-9214-0f03444481c0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:27.089622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f46ac99-0f51-4e26-9214-0f03444481c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:07:27.094418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-7f46ac99-0f51-4e26-9214-0f03444481c0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:07:27.094418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:07:27.094418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:07:27.094418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:07:27.094418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:07:27.094418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:07:27.094418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:07:27.094418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:07:27.094418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:07:27.094418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:07:27.094418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:07:27.094418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:07:27.094418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:07:27.094418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:07:27.094418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:07:27.094418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:07:27.095541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:07:27.095541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:07:27.095541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:07:27.095541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:07:27.095541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:07:27.095541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:07:27.095541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:07:27.095541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:07:27.095541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:07:27.095541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:07:27.095541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:07:27.095541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:07:27.095541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:07:27.095541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:07:27.095541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:07:27.095541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:07:27.097145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:07:27.097145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:07:27.097145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:07:27.097145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:07:27.097145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:07:27.097145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:07:27.097145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:07:27.097145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:07:27.097145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:07:27.097145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:07:27.097145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:07:27.097145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:07:27.097145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 569/1149] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:07:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:07:27.101875 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bfc1fc-b2d6-48e5-8a6e-213bef55cd1a None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:27.146424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:27.198169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bfc1fc-b2d6-48e5-8a6e-213bef55cd1a None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:27.230724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cc09a4-381f-4a46-9842-8fee8062e393 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:27.231371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cc09a4-381f-4a46-9842-8fee8062e393 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:27.231645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cc09a4-381f-4a46-9842-8fee8062e393 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:27.231910 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cc09a4-381f-4a46-9842-8fee8062e393 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:27.232553 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cc09a4-381f-4a46-9842-8fee8062e393 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:27.232848 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cc09a4-381f-4a46-9842-8fee8062e393 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:27.234185 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cc09a4-381f-4a46-9842-8fee8062e393 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:27.242548 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 581/1150] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:07:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:27.248360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f46ac99-0f51-4e26-9214-0f03444481c0 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:27.344788 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f46ac99-0f51-4e26-9214-0f03444481c0 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:27.378305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58b22d4-a17e-48e5-afc5-892861ffc5b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:27.378942 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58b22d4-a17e-48e5-afc5-892861ffc5b3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:27.379225 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58b22d4-a17e-48e5-afc5-892861ffc5b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:27.379499 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58b22d4-a17e-48e5-afc5-892861ffc5b3 None admin] PATH_INFO: `/v3/users/fffd672596964a97bfe3c947d0f5072a` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:27.380606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58b22d4-a17e-48e5-afc5-892861ffc5b3 None admin] RBAC: Authorizing `identity:get_user(user_id=fffd672596964a97bfe3c947d0f5072a)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:27.382018 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58b22d4-a17e-48e5-afc5-892861ffc5b3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:27.383589 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 570/1151] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:07:27 2022] GET /identity/v3/users/fffd672596964a97bfe3c947d0f5072a => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:27.389085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cc09a4-381f-4a46-9842-8fee8062e393 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:27.486654 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cc09a4-381f-4a46-9842-8fee8062e393 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:27.495115 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:27.519623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dee1337-3193-4fca-905f-d2aabac70385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:27.520244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dee1337-3193-4fca-905f-d2aabac70385 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:27.520522 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dee1337-3193-4fca-905f-d2aabac70385 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:27.520793 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dee1337-3193-4fca-905f-d2aabac70385 None admin] PATH_INFO: `/v3/projects/b86e433947634f47a189f296c7fe86ad` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:27.521890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dee1337-3193-4fca-905f-d2aabac70385 None admin] RBAC: Authorizing `identity:get_project(project_id=b86e433947634f47a189f296c7fe86ad)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:27.523348 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dee1337-3193-4fca-905f-d2aabac70385 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:27.524954 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 582/1152] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:07:27 2022] GET /identity/v3/projects/b86e433947634f47a189f296c7fe86ad => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:27.530746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58b22d4-a17e-48e5-afc5-892861ffc5b3 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:27.533585 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:27.633576 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58b22d4-a17e-48e5-afc5-892861ffc5b3 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:27.665748 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d557f1ba-adaf-4a04-9e76-94b3d2af1543 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:27.666395 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d557f1ba-adaf-4a04-9e76-94b3d2af1543 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:27.666675 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d557f1ba-adaf-4a04-9e76-94b3d2af1543 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:27.666980 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d557f1ba-adaf-4a04-9e76-94b3d2af1543 None admin] PATH_INFO: `/v3/projects/b86e433947634f47a189f296c7fe86ad/users/fffd672596964a97bfe3c947d0f5072a/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:27.668921 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d557f1ba-adaf-4a04-9e76-94b3d2af1543 None admin] RBAC: Authorizing `identity:create_grant(project_id=b86e433947634f47a189f296c7fe86ad, user_id=fffd672596964a97bfe3c947d0f5072a, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:27.670274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d557f1ba-adaf-4a04-9e76-94b3d2af1543 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:27.683592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 571/1153] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:07:27 2022] PUT /identity/v3/projects/b86e433947634f47a189f296c7fe86ad/users/fffd672596964a97bfe3c947d0f5072a/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:07:29.786842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bafc2e-8cf4-47b4-8d1d-3915f1aa7832 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:29.787396 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bafc2e-8cf4-47b4-8d1d-3915f1aa7832 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:29.787396 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bafc2e-8cf4-47b4-8d1d-3915f1aa7832 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:29.788743 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 583/1154] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:07:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:07:29.793903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf4b4ef-42df-4ea2-bc96-d4fcc93de178 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:29.794310 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf4b4ef-42df-4ea2-bc96-d4fcc93de178 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:29.794450 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf4b4ef-42df-4ea2-bc96-d4fcc93de178 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:29.813046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-baf4b4ef-42df-4ea2-bc96-d4fcc93de178 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:07:29.814220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-baf4b4ef-42df-4ea2-bc96-d4fcc93de178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:07:29.868357 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:29.879314 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 572/1155] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:07:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:07:30.148837 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96bafc2e-8cf4-47b4-8d1d-3915f1aa7832 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:30.248421 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96bafc2e-8cf4-47b4-8d1d-3915f1aa7832 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:30.279946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e778721-4590-4f18-ada9-721913ac2e80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:30.280558 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e778721-4590-4f18-ada9-721913ac2e80 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:30.280835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e778721-4590-4f18-ada9-721913ac2e80 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:30.281102 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e778721-4590-4f18-ada9-721913ac2e80 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:30.291807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e778721-4590-4f18-ada9-721913ac2e80 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:07:30.296513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-4e778721-4590-4f18-ada9-721913ac2e80 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:07:30.296513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:07:30.296513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:07:30.296513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:07:30.296513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:07:30.296513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:07:30.296513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:07:30.296513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:07:30.296513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:07:30.296513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:07:30.296513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:07:30.296513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:07:30.296513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:07:30.296513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:07:30.296513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:07:30.296513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:07:30.297608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:07:30.297608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:07:30.297608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:07:30.297608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:07:30.297608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:07:30.297608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:07:30.297608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:07:30.297608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:07:30.297608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:07:30.297608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:07:30.297608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:07:30.297608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:07:30.297608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:07:30.297608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:07:30.297608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:07:30.297608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:07:30.298658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:07:30.298658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:07:30.298658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:07:30.298658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:07:30.298658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:07:30.298658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:07:30.298658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:07:30.298658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:07:30.298658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:07:30.298658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:07:30.298658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:07:30.298658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:07:30.298658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 584/1156] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:07:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:07:30.303978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf4b4ef-42df-4ea2-bc96-d4fcc93de178 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:30.399084 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf4b4ef-42df-4ea2-bc96-d4fcc93de178 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:30.432525 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012954d9-b900-469a-b52d-c7f9d35b1fc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:30.433126 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012954d9-b900-469a-b52d-c7f9d35b1fc5 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:30.433402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012954d9-b900-469a-b52d-c7f9d35b1fc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:30.433669 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012954d9-b900-469a-b52d-c7f9d35b1fc5 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:30.434309 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012954d9-b900-469a-b52d-c7f9d35b1fc5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:30.434606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012954d9-b900-469a-b52d-c7f9d35b1fc5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:30.435987 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012954d9-b900-469a-b52d-c7f9d35b1fc5 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:30.443904 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 573/1157] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:07:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:30.449721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e778721-4590-4f18-ada9-721913ac2e80 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:30.484629 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:30.546776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e778721-4590-4f18-ada9-721913ac2e80 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:30.578602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e481fa9e-a957-4bac-abec-c91e9203c794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:30.579235 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e481fa9e-a957-4bac-abec-c91e9203c794 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:30.579515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e481fa9e-a957-4bac-abec-c91e9203c794 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:30.579786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e481fa9e-a957-4bac-abec-c91e9203c794 None admin] PATH_INFO: `/v3/users/fffd672596964a97bfe3c947d0f5072a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:30.581453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e481fa9e-a957-4bac-abec-c91e9203c794 None admin] RBAC: Authorizing `identity:get_user(user_id=fffd672596964a97bfe3c947d0f5072a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:30.582885 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e481fa9e-a957-4bac-abec-c91e9203c794 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:30.584413 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 585/1158] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:07:30 2022] GET /identity/v3/users/fffd672596964a97bfe3c947d0f5072a => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:30.590112 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012954d9-b900-469a-b52d-c7f9d35b1fc5 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:30.642859 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:30.686607 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012954d9-b900-469a-b52d-c7f9d35b1fc5 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:30.718770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0c39f6-0c16-446c-9e57-0ce9c90145fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:30.719388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0c39f6-0c16-446c-9e57-0ce9c90145fd None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:30.719665 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0c39f6-0c16-446c-9e57-0ce9c90145fd None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:30.719935 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0c39f6-0c16-446c-9e57-0ce9c90145fd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:30.783907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0c39f6-0c16-446c-9e57-0ce9c90145fd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:30.785313 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0c39f6-0c16-446c-9e57-0ce9c90145fd None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:30.836300 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:30.880000 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 574/1159] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:07:30 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 01:07:30.885853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e481fa9e-a957-4bac-abec-c91e9203c794 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:30.981452 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e481fa9e-a957-4bac-abec-c91e9203c794 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:30.983717 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:31.014337 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc930b85-5fd9-4573-a82e-f47d85071210 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:31.014979 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc930b85-5fd9-4573-a82e-f47d85071210 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:31.015255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc930b85-5fd9-4573-a82e-f47d85071210 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:31.015525 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc930b85-5fd9-4573-a82e-f47d85071210 None admin] PATH_INFO: `/v3/projects/b86e433947634f47a189f296c7fe86ad` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:31.016608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc930b85-5fd9-4573-a82e-f47d85071210 None admin] RBAC: Authorizing `identity:get_project(project_id=b86e433947634f47a189f296c7fe86ad)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:31.017997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc930b85-5fd9-4573-a82e-f47d85071210 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:31.019616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 586/1160] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:07:30 2022] GET /identity/v3/projects/b86e433947634f47a189f296c7fe86ad => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:31.025756 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0c39f6-0c16-446c-9e57-0ce9c90145fd None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:31.122764 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0c39f6-0c16-446c-9e57-0ce9c90145fd None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:31.155255 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a139edf-0a6d-48d0-a720-928c07f31cfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:31.155846 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a139edf-0a6d-48d0-a720-928c07f31cfb None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:31.156127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a139edf-0a6d-48d0-a720-928c07f31cfb None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:31.156399 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a139edf-0a6d-48d0-a720-928c07f31cfb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:31.157111 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a139edf-0a6d-48d0-a720-928c07f31cfb None admin] RBAC: Adding query filter params (role.id=66a36c858aab40688080eed6ceace3a0, scope.project.id=b86e433947634f47a189f296c7fe86ad, user.id=fffd672596964a97bfe3c947d0f5072a) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:31.157419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a139edf-0a6d-48d0-a720-928c07f31cfb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:31.159380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a139edf-0a6d-48d0-a720-928c07f31cfb None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:31.166008 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 575/1161] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:07:31 2022] GET /identity/v3/role_assignments?user.id=fffd672596964a97bfe3c947d0f5072a&scope.project.id=b86e433947634f47a189f296c7fe86ad&role.id=66a36c858aab40688080eed6ceace3a0 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:33.313115 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc049353-b28b-49e3-a0af-22b4cf751ef7 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:33.313667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc049353-b28b-49e3-a0af-22b4cf751ef7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:33.313667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc049353-b28b-49e3-a0af-22b4cf751ef7 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:33.315541 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 587/1162] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:07:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:07:33.320715 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad7f1f3-b980-444b-8923-edf241047031 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:33.321135 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad7f1f3-b980-444b-8923-edf241047031 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:33.321240 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad7f1f3-b980-444b-8923-edf241047031 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:33.339460 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-aad7f1f3-b980-444b-8923-edf241047031 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:07:33.340624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-aad7f1f3-b980-444b-8923-edf241047031 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:07:33.353177 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:33.405585 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 576/1163] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:07:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:07:33.670586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc049353-b28b-49e3-a0af-22b4cf751ef7 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:33.769160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc049353-b28b-49e3-a0af-22b4cf751ef7 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:33.802200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d2e457-dcab-449e-9cfe-1c9293e21b4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:33.802788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d2e457-dcab-449e-9cfe-1c9293e21b4e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:33.803090 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d2e457-dcab-449e-9cfe-1c9293e21b4e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:33.803356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d2e457-dcab-449e-9cfe-1c9293e21b4e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:33.842048 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:33.868413 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d2e457-dcab-449e-9cfe-1c9293e21b4e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:33.869832 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d2e457-dcab-449e-9cfe-1c9293e21b4e None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:33.962832 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 588/1164] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:07:33 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 01:07:33.968770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad7f1f3-b980-444b-8923-edf241047031 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:34.062259 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad7f1f3-b980-444b-8923-edf241047031 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:34.093826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e8ab6a-4285-4fff-9f99-95bfc0b4129a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:34.094432 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e8ab6a-4285-4fff-9f99-95bfc0b4129a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:34.094704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e8ab6a-4285-4fff-9f99-95bfc0b4129a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:34.094996 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e8ab6a-4285-4fff-9f99-95bfc0b4129a None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:34.101776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e8ab6a-4285-4fff-9f99-95bfc0b4129a None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:34.103211 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e8ab6a-4285-4fff-9f99-95bfc0b4129a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:34.114088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-71e8ab6a-4285-4fff-9f99-95bfc0b4129a None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 13 01:07:34.114088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:07:34.114088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 13 01:07:34.114088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 13 01:07:34.114088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:07:34.114088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:07:34.114088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:07:34.114088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:07:34.114088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 13 01:07:34.114088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:07:34.114088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:07:34.114088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:07:34.114088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:07:34.114088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:07:34.114088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:07:34.114088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:07:34.114088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:07:34.119927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:07:34.119927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:07:34.119927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:07:34.119927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:07:34.119927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:07:34.119927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:07:34.119927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:07:34.119927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:07:34.119927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:07:34.119927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:07:34.119927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:07:34.119927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:07:34.119927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:07:34.119927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:07:34.119927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:07:34.119927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:07:34.120969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:07:34.120969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:07:34.120969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:07:34.120969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:07:34.120969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:07:34.120969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:07:34.120969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:07:34.120969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:07:34.120969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:07:34.120969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:07:34.120969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 13 01:07:34.120969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:07:34.120969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 13 01:07:34.120969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:07:34.120969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 577/1165] 10.222.0.15 () {56 vars in 1117 bytes} [Tue Sep 13 01:07:33 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:07:34.121960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d2e457-dcab-449e-9cfe-1c9293e21b4e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:34.187870 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:34.216202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d2e457-dcab-449e-9cfe-1c9293e21b4e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:34.248153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41d73ad-8349-4d67-ac62-8cf13eed943d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:34.248740 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41d73ad-8349-4d67-ac62-8cf13eed943d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:34.249012 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41d73ad-8349-4d67-ac62-8cf13eed943d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:34.249277 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41d73ad-8349-4d67-ac62-8cf13eed943d None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:34.249897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41d73ad-8349-4d67-ac62-8cf13eed943d None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:34.250192 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41d73ad-8349-4d67-ac62-8cf13eed943d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:34.251577 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41d73ad-8349-4d67-ac62-8cf13eed943d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:34.258901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 589/1166] 10.222.0.15 () {56 vars in 1115 bytes} [Tue Sep 13 01:07:34 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:36.338055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fb2f93-73c9-44a2-a420-4eea2991d37d None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:36.338677 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fb2f93-73c9-44a2-a420-4eea2991d37d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:36.338677 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fb2f93-73c9-44a2-a420-4eea2991d37d None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:36.339993 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 578/1167] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:07:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:07:36.345304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed53bd6-3caf-4d86-8396-fc941f15a80a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:36.345699 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed53bd6-3caf-4d86-8396-fc941f15a80a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:36.345853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed53bd6-3caf-4d86-8396-fc941f15a80a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:36.364617 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8ed53bd6-3caf-4d86-8396-fc941f15a80a None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:07:36.365767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8ed53bd6-3caf-4d86-8396-fc941f15a80a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:07:36.429527 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 590/1168] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:07:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:07:36.701550 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4fb2f93-73c9-44a2-a420-4eea2991d37d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:36.713170 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:36.800445 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4fb2f93-73c9-44a2-a420-4eea2991d37d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:36.832541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b37c342-c129-4a18-a7f9-fa0fdb6a317e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:36.833132 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b37c342-c129-4a18-a7f9-fa0fdb6a317e None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:36.833412 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b37c342-c129-4a18-a7f9-fa0fdb6a317e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:36.833683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b37c342-c129-4a18-a7f9-fa0fdb6a317e None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:36.834343 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b37c342-c129-4a18-a7f9-fa0fdb6a317e None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:36.835716 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b37c342-c129-4a18-a7f9-fa0fdb6a317e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:36.851528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 579/1169] 10.222.0.15 () {60 vars in 1138 bytes} [Tue Sep 13 01:07:36 2022] POST /identity/v3/domains => generated 267 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:07:38.988718 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96efd9de-3b30-4dba-8e41-b0d3d3e2e062 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:38.989249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96efd9de-3b30-4dba-8e41-b0d3d3e2e062 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:38.989249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96efd9de-3b30-4dba-8e41-b0d3d3e2e062 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:38.990586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 591/1170] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:07:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:07:38.995948 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5edc623-847e-4770-83dc-1b77fbeed04b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:38.996338 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5edc623-847e-4770-83dc-1b77fbeed04b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:38.996502 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5edc623-847e-4770-83dc-1b77fbeed04b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:39.014690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c5edc623-847e-4770-83dc-1b77fbeed04b None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:07:39.015907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c5edc623-847e-4770-83dc-1b77fbeed04b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:07:39.075666 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:39.082232 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 580/1171] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:07:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:07:39.347523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96efd9de-3b30-4dba-8e41-b0d3d3e2e062 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:39.411484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:39.446490 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96efd9de-3b30-4dba-8e41-b0d3d3e2e062 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:39.479410 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9c40f7-6ad8-4ac8-8bf7-a060079b1959 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:39.480021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9c40f7-6ad8-4ac8-8bf7-a060079b1959 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:39.480296 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9c40f7-6ad8-4ac8-8bf7-a060079b1959 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:39.480564 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9c40f7-6ad8-4ac8-8bf7-a060079b1959 None admin] PATH_INFO: `/v3/domains/e5d8c28c004f4ec7b524136371bc8259` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:39.490429 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9c40f7-6ad8-4ac8-8bf7-a060079b1959 None admin] RBAC: Authorizing `identity:get_domain(domain_id=e5d8c28c004f4ec7b524136371bc8259)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:39.491880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9c40f7-6ad8-4ac8-8bf7-a060079b1959 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:39.493430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 592/1172] 10.222.0.15 () {56 vars in 1183 bytes} [Tue Sep 13 01:07:39 2022] GET /identity/v3/domains/e5d8c28c004f4ec7b524136371bc8259 => generated 267 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:07:39.499346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5edc623-847e-4770-83dc-1b77fbeed04b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:39.593622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5edc623-847e-4770-83dc-1b77fbeed04b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:39.625848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa47029-d2ec-4144-9c73-f6a6dab26768 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:39.626390 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa47029-d2ec-4144-9c73-f6a6dab26768 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:39.626664 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa47029-d2ec-4144-9c73-f6a6dab26768 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:39.626959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa47029-d2ec-4144-9c73-f6a6dab26768 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:39.627602 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa47029-d2ec-4144-9c73-f6a6dab26768 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:39.629014 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa47029-d2ec-4144-9c73-f6a6dab26768 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:39.655131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 581/1173] 10.222.0.15 () {60 vars in 1141 bytes} [Tue Sep 13 01:07:39 2022] POST /identity/v3/projects => generated 361 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:07:41.881453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c6731e-b092-40a5-b1c2-4e50db7cde87 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:41.882018 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c6731e-b092-40a5-b1c2-4e50db7cde87 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:41.882018 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c6731e-b092-40a5-b1c2-4e50db7cde87 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:41.883360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 593/1174] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:07:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:07:41.888650 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922094c2-5fbd-4a4c-bf60-4453be1c92e9 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:41.889049 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922094c2-5fbd-4a4c-bf60-4453be1c92e9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:41.889211 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922094c2-5fbd-4a4c-bf60-4453be1c92e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:41.907081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-922094c2-5fbd-4a4c-bf60-4453be1c92e9 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:07:41.908294 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-922094c2-5fbd-4a4c-bf60-4453be1c92e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:07:41.954348 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:41.975765 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 582/1175] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:07:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:07:42.248163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c6731e-b092-40a5-b1c2-4e50db7cde87 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:42.347267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c6731e-b092-40a5-b1c2-4e50db7cde87 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:42.371958 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:42.380853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36418514-f44b-4d9b-9ee2-54efecc66c63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:42.381465 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36418514-f44b-4d9b-9ee2-54efecc66c63 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:42.381740 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36418514-f44b-4d9b-9ee2-54efecc66c63 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:42.382009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36418514-f44b-4d9b-9ee2-54efecc66c63 None admin] PATH_INFO: `/v3/domains/e5d8c28c004f4ec7b524136371bc8259` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:42.383121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36418514-f44b-4d9b-9ee2-54efecc66c63 None admin] RBAC: Authorizing `identity:get_domain(domain_id=e5d8c28c004f4ec7b524136371bc8259)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:42.384525 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36418514-f44b-4d9b-9ee2-54efecc66c63 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:42.386052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 594/1176] 10.222.0.15 () {56 vars in 1183 bytes} [Tue Sep 13 01:07:42 2022] GET /identity/v3/domains/e5d8c28c004f4ec7b524136371bc8259 => generated 267 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:42.391710 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922094c2-5fbd-4a4c-bf60-4453be1c92e9 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:42.487250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922094c2-5fbd-4a4c-bf60-4453be1c92e9 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:42.519289 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efbbd15-3ed3-47c1-8800-95c2be83fd74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:42.519825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efbbd15-3ed3-47c1-8800-95c2be83fd74 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:42.520098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efbbd15-3ed3-47c1-8800-95c2be83fd74 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:42.520363 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efbbd15-3ed3-47c1-8800-95c2be83fd74 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:42.520989 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efbbd15-3ed3-47c1-8800-95c2be83fd74 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:42.522388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efbbd15-3ed3-47c1-8800-95c2be83fd74 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:42.543041 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 583/1177] 10.222.0.15 () {60 vars in 1132 bytes} [Tue Sep 13 01:07:42 2022] POST /identity/v3/users => generated 311 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:07:44.703978 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afee9be6-2d9b-48fd-9232-04d1b189b528 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:44.704472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afee9be6-2d9b-48fd-9232-04d1b189b528 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:44.704472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afee9be6-2d9b-48fd-9232-04d1b189b528 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:44.706275 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 595/1178] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:07:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:07:44.711438 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fd7ff5-3ec4-4387-b224-95cde23e53c9 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:44.711824 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fd7ff5-3ec4-4387-b224-95cde23e53c9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:44.711986 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fd7ff5-3ec4-4387-b224-95cde23e53c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:44.730220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-99fd7ff5-3ec4-4387-b224-95cde23e53c9 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:07:44.731419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-99fd7ff5-3ec4-4387-b224-95cde23e53c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:07:44.758769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:44.796876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 584/1179] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:07:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:07:45.063933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afee9be6-2d9b-48fd-9232-04d1b189b528 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:45.163411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afee9be6-2d9b-48fd-9232-04d1b189b528 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:45.196154 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6a3b92-9a69-418b-9986-2d5175f4df0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:45.196770 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6a3b92-9a69-418b-9986-2d5175f4df0b None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:45.197050 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6a3b92-9a69-418b-9986-2d5175f4df0b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:45.197323 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6a3b92-9a69-418b-9986-2d5175f4df0b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:45.207508 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6a3b92-9a69-418b-9986-2d5175f4df0b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:07:45.212225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-3b6a3b92-9a69-418b-9986-2d5175f4df0b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:07:45.212225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:07:45.212225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:07:45.212225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:07:45.212225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:07:45.212225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:07:45.212225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:07:45.212225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:07:45.212225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:07:45.212225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:07:45.212225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:07:45.212225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:07:45.212225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:07:45.212225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:07:45.212225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:07:45.212225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:07:45.213376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:07:45.213376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:07:45.213376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:07:45.213376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:07:45.213376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:07:45.213376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:07:45.213376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:07:45.213376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:07:45.213376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:07:45.213376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:07:45.213376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:07:45.213376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:07:45.213376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:07:45.213376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:07:45.213376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:07:45.213376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:07:45.214602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:07:45.214602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:07:45.214602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:07:45.214602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:07:45.214602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:07:45.214602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:07:45.214602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:07:45.214602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:07:45.214602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:07:45.214602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:07:45.214602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:07:45.214602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:07:45.214602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 596/1180] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:07:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:07:45.219256 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fd7ff5-3ec4-4387-b224-95cde23e53c9 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:45.314424 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fd7ff5-3ec4-4387-b224-95cde23e53c9 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:45.347262 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad3e1e8-01d3-42a3-b76d-0c552a0cc5c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:45.347862 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad3e1e8-01d3-42a3-b76d-0c552a0cc5c3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:45.348135 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad3e1e8-01d3-42a3-b76d-0c552a0cc5c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:45.348404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad3e1e8-01d3-42a3-b76d-0c552a0cc5c3 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:45.349052 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad3e1e8-01d3-42a3-b76d-0c552a0cc5c3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:45.349352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad3e1e8-01d3-42a3-b76d-0c552a0cc5c3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:45.350697 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad3e1e8-01d3-42a3-b76d-0c552a0cc5c3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:45.358476 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 585/1181] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:07:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:45.365049 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6a3b92-9a69-418b-9986-2d5175f4df0b None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:45.418025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:45.462022 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6a3b92-9a69-418b-9986-2d5175f4df0b None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:45.493893 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51b6383-6251-4dd4-a059-49e06edc3859 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:45.494504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51b6383-6251-4dd4-a059-49e06edc3859 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:45.494779 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51b6383-6251-4dd4-a059-49e06edc3859 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:45.495069 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51b6383-6251-4dd4-a059-49e06edc3859 None admin] PATH_INFO: `/v3/users/e08503913df84112bef8b346631f4396` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:45.507447 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51b6383-6251-4dd4-a059-49e06edc3859 None admin] RBAC: Authorizing `identity:get_user(user_id=e08503913df84112bef8b346631f4396)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:45.508850 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51b6383-6251-4dd4-a059-49e06edc3859 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:45.510391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 597/1182] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:07:45 2022] GET /identity/v3/users/e08503913df84112bef8b346631f4396 => generated 311 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:45.516233 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad3e1e8-01d3-42a3-b76d-0c552a0cc5c3 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:45.544699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:45.613945 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad3e1e8-01d3-42a3-b76d-0c552a0cc5c3 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:45.646081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac22e00-b01d-473d-ab8d-ad68a4d2459f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:45.646671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac22e00-b01d-473d-ab8d-ad68a4d2459f None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:45.646983 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac22e00-b01d-473d-ab8d-ad68a4d2459f None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:45.647255 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac22e00-b01d-473d-ab8d-ad68a4d2459f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:45.712202 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac22e00-b01d-473d-ab8d-ad68a4d2459f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:45.713640 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac22e00-b01d-473d-ab8d-ad68a4d2459f None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:45.739607 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:45.808333 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 586/1183] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:07:45 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 01:07:45.815017 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51b6383-6251-4dd4-a059-49e06edc3859 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:45.911178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51b6383-6251-4dd4-a059-49e06edc3859 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:45.923550 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:45.944728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac1ecf2-61f2-4ba2-be4d-acc646dc27be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:45.945343 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac1ecf2-61f2-4ba2-be4d-acc646dc27be None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:45.945620 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac1ecf2-61f2-4ba2-be4d-acc646dc27be None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:45.945892 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac1ecf2-61f2-4ba2-be4d-acc646dc27be None admin] PATH_INFO: `/v3/projects/185cccf6d1114c5cb5528980e58367ec` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:45.947009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac1ecf2-61f2-4ba2-be4d-acc646dc27be None admin] RBAC: Authorizing `identity:get_project(project_id=185cccf6d1114c5cb5528980e58367ec)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:45.948416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac1ecf2-61f2-4ba2-be4d-acc646dc27be None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:45.950001 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 598/1184] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:07:45 2022] GET /identity/v3/projects/185cccf6d1114c5cb5528980e58367ec => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:45.955996 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac22e00-b01d-473d-ab8d-ad68a4d2459f None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:46.051055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac22e00-b01d-473d-ab8d-ad68a4d2459f None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:46.076050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:46.083857 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cbf816-f214-4ec0-9e14-03c18d85b3a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:46.084447 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cbf816-f214-4ec0-9e14-03c18d85b3a0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:46.084723 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cbf816-f214-4ec0-9e14-03c18d85b3a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:46.084988 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cbf816-f214-4ec0-9e14-03c18d85b3a0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:46.085688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cbf816-f214-4ec0-9e14-03c18d85b3a0 None admin] RBAC: Adding query filter params (role.id=66a36c858aab40688080eed6ceace3a0, scope.project.id=185cccf6d1114c5cb5528980e58367ec, user.id=e08503913df84112bef8b346631f4396) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:46.085990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cbf816-f214-4ec0-9e14-03c18d85b3a0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:46.087420 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cbf816-f214-4ec0-9e14-03c18d85b3a0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:46.092552 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 587/1185] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:07:45 2022] GET /identity/v3/role_assignments?user.id=e08503913df84112bef8b346631f4396&scope.project.id=185cccf6d1114c5cb5528980e58367ec&role.id=66a36c858aab40688080eed6ceace3a0 => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:48.189586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9a1b72-38c3-4d4d-b563-170625e2c46d None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:48.190123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9a1b72-38c3-4d4d-b563-170625e2c46d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:48.190123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9a1b72-38c3-4d4d-b563-170625e2c46d None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:48.191487 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 599/1186] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:07:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:07:48.196508 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2862fe-16ad-4bef-99fb-f877907d1258 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:48.196876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2862fe-16ad-4bef-99fb-f877907d1258 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:48.197053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2862fe-16ad-4bef-99fb-f877907d1258 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:48.215235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6d2862fe-16ad-4bef-99fb-f877907d1258 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:07:48.216403 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6d2862fe-16ad-4bef-99fb-f877907d1258 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:07:48.279437 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 588/1187] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:07:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:07:48.550025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9a1b72-38c3-4d4d-b563-170625e2c46d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:48.648038 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9a1b72-38c3-4d4d-b563-170625e2c46d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:48.680403 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd38483-0c47-4a3e-a17c-6db7337d4fd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:48.681016 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd38483-0c47-4a3e-a17c-6db7337d4fd4 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:48.681295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd38483-0c47-4a3e-a17c-6db7337d4fd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:48.681563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd38483-0c47-4a3e-a17c-6db7337d4fd4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:48.692045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd38483-0c47-4a3e-a17c-6db7337d4fd4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:07:48.696748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-fcd38483-0c47-4a3e-a17c-6db7337d4fd4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:07:48.696748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:07:48.696748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:07:48.696748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:07:48.696748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:07:48.696748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:07:48.696748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:07:48.696748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:07:48.696748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:07:48.696748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:07:48.696748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:07:48.696748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:07:48.696748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:07:48.696748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:07:48.696748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:07:48.696748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:07:48.697821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:07:48.697821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:07:48.697821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:07:48.697821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:07:48.697821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:07:48.697821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:07:48.697821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:07:48.697821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:07:48.697821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:07:48.697821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:07:48.697821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:07:48.697821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:07:48.697821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:07:48.697821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:07:48.697821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:07:48.697821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:07:48.699178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:07:48.699178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:07:48.699178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:07:48.699178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:07:48.699178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:07:48.699178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:07:48.699178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:07:48.699178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:07:48.699178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:07:48.699178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:07:48.699178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:07:48.699178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:07:48.699178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 600/1188] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:07:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:07:48.703823 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2862fe-16ad-4bef-99fb-f877907d1258 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:48.796438 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2862fe-16ad-4bef-99fb-f877907d1258 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:48.818844 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:48.830181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161fc721-c252-4e62-9074-4da49f7bf942 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:48.830772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161fc721-c252-4e62-9074-4da49f7bf942 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:48.831072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161fc721-c252-4e62-9074-4da49f7bf942 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:48.831340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161fc721-c252-4e62-9074-4da49f7bf942 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:48.831985 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161fc721-c252-4e62-9074-4da49f7bf942 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:48.832282 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161fc721-c252-4e62-9074-4da49f7bf942 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:48.833624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161fc721-c252-4e62-9074-4da49f7bf942 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:48.844293 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 589/1189] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:07:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:48.851268 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd38483-0c47-4a3e-a17c-6db7337d4fd4 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:48.892839 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:48.948184 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd38483-0c47-4a3e-a17c-6db7337d4fd4 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:48.980671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee82a65f-fd20-415f-b69e-d11102578692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:48.981282 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee82a65f-fd20-415f-b69e-d11102578692 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:48.981556 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee82a65f-fd20-415f-b69e-d11102578692 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:48.981824 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee82a65f-fd20-415f-b69e-d11102578692 None admin] PATH_INFO: `/v3/users/e08503913df84112bef8b346631f4396` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:48.982933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee82a65f-fd20-415f-b69e-d11102578692 None admin] RBAC: Authorizing `identity:get_user(user_id=e08503913df84112bef8b346631f4396)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:48.984368 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee82a65f-fd20-415f-b69e-d11102578692 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:48.986480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 601/1190] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:07:48 2022] GET /identity/v3/users/e08503913df84112bef8b346631f4396 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:48.992276 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161fc721-c252-4e62-9074-4da49f7bf942 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:49.088032 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161fc721-c252-4e62-9074-4da49f7bf942 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:49.120787 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19057b1-ad17-47f4-bbd7-51865d0ba1b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:49.121397 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19057b1-ad17-47f4-bbd7-51865d0ba1b2 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:49.121677 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19057b1-ad17-47f4-bbd7-51865d0ba1b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:49.121950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19057b1-ad17-47f4-bbd7-51865d0ba1b2 None admin] PATH_INFO: `/v3/projects/185cccf6d1114c5cb5528980e58367ec` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:49.123075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19057b1-ad17-47f4-bbd7-51865d0ba1b2 None admin] RBAC: Authorizing `identity:get_project(project_id=185cccf6d1114c5cb5528980e58367ec)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:49.124486 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19057b1-ad17-47f4-bbd7-51865d0ba1b2 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:49.126087 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 590/1191] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:07:48 2022] GET /identity/v3/projects/185cccf6d1114c5cb5528980e58367ec => generated 361 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:49.131912 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee82a65f-fd20-415f-b69e-d11102578692 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:49.229529 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee82a65f-fd20-415f-b69e-d11102578692 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:49.238473 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:49.262258 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420bf418-e4ac-476a-be64-7978e1efcd3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:49.262923 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420bf418-e4ac-476a-be64-7978e1efcd3b None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:49.263197 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420bf418-e4ac-476a-be64-7978e1efcd3b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:49.263463 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420bf418-e4ac-476a-be64-7978e1efcd3b None admin] PATH_INFO: `/v3/projects/185cccf6d1114c5cb5528980e58367ec/users/e08503913df84112bef8b346631f4396/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:49.265296 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420bf418-e4ac-476a-be64-7978e1efcd3b None admin] RBAC: Authorizing `identity:create_grant(project_id=185cccf6d1114c5cb5528980e58367ec, user_id=e08503913df84112bef8b346631f4396, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:49.266696 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420bf418-e4ac-476a-be64-7978e1efcd3b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:49.279206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 602/1192] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:07:49 2022] PUT /identity/v3/projects/185cccf6d1114c5cb5528980e58367ec/users/e08503913df84112bef8b346631f4396/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:07:51.377805 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4228ccd-43ae-4dce-9385-cd11bd1d3da2 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:51.378332 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4228ccd-43ae-4dce-9385-cd11bd1d3da2 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:51.378332 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4228ccd-43ae-4dce-9385-cd11bd1d3da2 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:51.379702 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 591/1193] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:07:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:07:51.384977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8132d022-65b6-4966-93c6-33ad4b8f5d1b None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:51.385363 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8132d022-65b6-4966-93c6-33ad4b8f5d1b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:51.385525 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8132d022-65b6-4966-93c6-33ad4b8f5d1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:51.403979 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8132d022-65b6-4966-93c6-33ad4b8f5d1b None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:07:51.405156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8132d022-65b6-4966-93c6-33ad4b8f5d1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:07:51.469699 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 603/1194] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:07:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:07:51.733325 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4228ccd-43ae-4dce-9385-cd11bd1d3da2 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:51.785666 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:51.832310 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4228ccd-43ae-4dce-9385-cd11bd1d3da2 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:51.863813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218283f5-ef4a-4cfb-8e9f-53d93a8d936e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:51.864419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218283f5-ef4a-4cfb-8e9f-53d93a8d936e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:51.864694 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218283f5-ef4a-4cfb-8e9f-53d93a8d936e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:51.864961 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218283f5-ef4a-4cfb-8e9f-53d93a8d936e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:51.874511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218283f5-ef4a-4cfb-8e9f-53d93a8d936e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:07:51.879740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-218283f5-ef4a-4cfb-8e9f-53d93a8d936e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:07:51.879740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:07:51.879740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:07:51.879740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:07:51.879740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:07:51.879740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:07:51.879740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:07:51.879740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:07:51.879740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:07:51.879740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:07:51.879740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:07:51.879740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:07:51.879740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:07:51.879740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:07:51.879740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:07:51.879740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:07:51.880836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:07:51.880836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:07:51.880836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:07:51.880836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:07:51.880836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:07:51.880836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:07:51.880836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:07:51.880836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:07:51.880836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:07:51.880836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:07:51.880836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:07:51.880836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:07:51.880836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:07:51.880836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:07:51.880836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:07:51.880836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:07:51.881726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:07:51.881726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:07:51.881726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:07:51.881726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:07:51.881726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:07:51.881726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:07:51.881726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:07:51.881726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:07:51.881726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:07:51.881726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:07:51.881726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:07:51.881726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:07:51.881726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 592/1195] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:07:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:07:51.886649 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8132d022-65b6-4966-93c6-33ad4b8f5d1b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:51.990470 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:51.994423 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8132d022-65b6-4966-93c6-33ad4b8f5d1b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:52.028610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8314c433-6bff-4195-9877-f39fffa7eaab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:52.029199 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8314c433-6bff-4195-9877-f39fffa7eaab None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:52.029473 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8314c433-6bff-4195-9877-f39fffa7eaab None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:52.029741 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8314c433-6bff-4195-9877-f39fffa7eaab None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:52.030384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8314c433-6bff-4195-9877-f39fffa7eaab None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:52.030680 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8314c433-6bff-4195-9877-f39fffa7eaab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:52.032054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8314c433-6bff-4195-9877-f39fffa7eaab None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:52.039757 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 604/1196] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:07:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:52.045848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218283f5-ef4a-4cfb-8e9f-53d93a8d936e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:52.143068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218283f5-ef4a-4cfb-8e9f-53d93a8d936e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:52.155495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:52.176483 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2be0e45-beaa-4b71-9c62-d9077e61c50e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:52.177092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2be0e45-beaa-4b71-9c62-d9077e61c50e None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:52.177368 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2be0e45-beaa-4b71-9c62-d9077e61c50e None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:52.177635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2be0e45-beaa-4b71-9c62-d9077e61c50e None admin] PATH_INFO: `/v3/users/e08503913df84112bef8b346631f4396` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:52.178723 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2be0e45-beaa-4b71-9c62-d9077e61c50e None admin] RBAC: Authorizing `identity:get_user(user_id=e08503913df84112bef8b346631f4396)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:52.180160 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2be0e45-beaa-4b71-9c62-d9077e61c50e None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:52.181692 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 593/1197] 10.222.0.15 () {56 vars in 1177 bytes} [Tue Sep 13 01:07:52 2022] GET /identity/v3/users/e08503913df84112bef8b346631f4396 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:52.187637 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8314c433-6bff-4195-9877-f39fffa7eaab None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:52.289524 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8314c433-6bff-4195-9877-f39fffa7eaab None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:52.331941 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f42cb14-3678-4900-a20e-632de0553cb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:52.332545 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f42cb14-3678-4900-a20e-632de0553cb6 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:52.332830 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f42cb14-3678-4900-a20e-632de0553cb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:52.333103 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f42cb14-3678-4900-a20e-632de0553cb6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:52.359789 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:52.399248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f42cb14-3678-4900-a20e-632de0553cb6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:52.400678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f42cb14-3678-4900-a20e-632de0553cb6 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:52.495580 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 605/1198] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:07:52 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 01:07:52.502218 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2be0e45-beaa-4b71-9c62-d9077e61c50e None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:52.597721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2be0e45-beaa-4b71-9c62-d9077e61c50e None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:52.630709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca589ff-e6ac-4bb7-99ef-10876d7652dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:52.631357 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca589ff-e6ac-4bb7-99ef-10876d7652dc None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:52.631637 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca589ff-e6ac-4bb7-99ef-10876d7652dc None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:52.631900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca589ff-e6ac-4bb7-99ef-10876d7652dc None admin] PATH_INFO: `/v3/projects/185cccf6d1114c5cb5528980e58367ec` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:52.632990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca589ff-e6ac-4bb7-99ef-10876d7652dc None admin] RBAC: Authorizing `identity:get_project(project_id=185cccf6d1114c5cb5528980e58367ec)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:52.634393 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca589ff-e6ac-4bb7-99ef-10876d7652dc None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:52.636036 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 594/1199] 10.222.0.15 () {56 vars in 1186 bytes} [Tue Sep 13 01:07:52 2022] GET /identity/v3/projects/185cccf6d1114c5cb5528980e58367ec => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:07:52.642262 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f42cb14-3678-4900-a20e-632de0553cb6 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:07:52.699530 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:07:52.739600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f42cb14-3678-4900-a20e-632de0553cb6 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:07:52.771850 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e649702-93ba-408d-9288-d6966d293fa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:07:52.772440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e649702-93ba-408d-9288-d6966d293fa1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:07:52.772718 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e649702-93ba-408d-9288-d6966d293fa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:07:52.772991 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e649702-93ba-408d-9288-d6966d293fa1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:07:52.773700 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e649702-93ba-408d-9288-d6966d293fa1 None admin] RBAC: Adding query filter params (role.id=66a36c858aab40688080eed6ceace3a0, scope.project.id=185cccf6d1114c5cb5528980e58367ec, user.id=e08503913df84112bef8b346631f4396) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:07:52.774004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e649702-93ba-408d-9288-d6966d293fa1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:07:52.775439 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e649702-93ba-408d-9288-d6966d293fa1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:07:52.780966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 606/1200] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:07:52 2022] GET /identity/v3/role_assignments?user.id=e08503913df84112bef8b346631f4396&scope.project.id=185cccf6d1114c5cb5528980e58367ec&role.id=66a36c858aab40688080eed6ceace3a0 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:08:02.626540 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=76 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:02.778450 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=38 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:48.394238 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142f8070-20f3-4931-8ca7-6ed55fbebadc None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:48.394971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142f8070-20f3-4931-8ca7-6ed55fbebadc None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:48.394971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142f8070-20f3-4931-8ca7-6ed55fbebadc None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:48.396312 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 595/1201] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:08:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:08:48.401829 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e6e9b3-139a-4eab-b78b-a69913c3cf68 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:48.401829 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e6e9b3-139a-4eab-b78b-a69913c3cf68 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:48.405470 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e6e9b3-139a-4eab-b78b-a69913c3cf68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:48.421205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-00e6e9b3-139a-4eab-b78b-a69913c3cf68 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:08:48.422514 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-00e6e9b3-139a-4eab-b78b-a69913c3cf68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:08:48.508772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 607/1202] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:08:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:08:48.792122 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142f8070-20f3-4931-8ca7-6ed55fbebadc None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:48.929150 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142f8070-20f3-4931-8ca7-6ed55fbebadc None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:48.971182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f28ce9e-6ded-42b4-810c-063b865a59d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:48.972016 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f28ce9e-6ded-42b4-810c-063b865a59d3 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:48.972298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f28ce9e-6ded-42b4-810c-063b865a59d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:48.972573 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f28ce9e-6ded-42b4-810c-063b865a59d3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:48.983797 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f28ce9e-6ded-42b4-810c-063b865a59d3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:48.985235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f28ce9e-6ded-42b4-810c-063b865a59d3 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:49.000500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-6f28ce9e-6ded-42b4-810c-063b865a59d3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:08:49.000500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:08:49.000500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:08:49.000500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:08:49.000500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:08:49.000500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:08:49.000500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:08:49.000500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:08:49.000500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:08:49.000500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:08:49.000500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:08:49.000500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:08:49.000500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:08:49.000500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:08:49.000500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:08:49.000500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:08:49.001729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:08:49.001729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:08:49.001729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:08:49.001729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:08:49.001729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:08:49.001729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:08:49.001729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:08:49.001729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:08:49.001729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:08:49.001729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:08:49.001729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:08:49.001729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:08:49.001729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:08:49.001729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:08:49.001729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:08:49.001729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:08:49.001729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:08:49.002832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:08:49.002832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:08:49.002832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:08:49.002832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 596/1203] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:08:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:08:49.007753 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e6e9b3-139a-4eab-b78b-a69913c3cf68 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:49.130007 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e6e9b3-139a-4eab-b78b-a69913c3cf68 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:49.163766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:49.172955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb235aa-604e-4fd3-81cb-88870ed6b074 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:49.173580 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb235aa-604e-4fd3-81cb-88870ed6b074 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:49.173862 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb235aa-604e-4fd3-81cb-88870ed6b074 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:49.174136 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb235aa-604e-4fd3-81cb-88870ed6b074 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:49.174798 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb235aa-604e-4fd3-81cb-88870ed6b074 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:08:49.175134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb235aa-604e-4fd3-81cb-88870ed6b074 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:49.181818 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb235aa-604e-4fd3-81cb-88870ed6b074 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:49.190148 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 608/1204] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:08:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:08:49.195454 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f28ce9e-6ded-42b4-810c-063b865a59d3 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:49.283015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:49.345931 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f28ce9e-6ded-42b4-810c-063b865a59d3 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:49.389488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a519828a-a4f5-44df-8c63-d1f0a08f9075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:49.390082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a519828a-a4f5-44df-8c63-d1f0a08f9075 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:49.390419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a519828a-a4f5-44df-8c63-d1f0a08f9075 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:49.390677 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a519828a-a4f5-44df-8c63-d1f0a08f9075 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:49.392826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a519828a-a4f5-44df-8c63-d1f0a08f9075 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:49.392973 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a519828a-a4f5-44df-8c63-d1f0a08f9075 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:49.420283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 597/1205] 10.222.0.15 () {60 vars in 1132 bytes} [Tue Sep 13 01:08:49 2022] POST /identity/v3/users => generated 251 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:08:51.813408 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0082edbc-c15b-43b4-ab1e-49e8a98e2a28 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:51.813983 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0082edbc-c15b-43b4-ab1e-49e8a98e2a28 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:51.813983 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0082edbc-c15b-43b4-ab1e-49e8a98e2a28 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:51.815545 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 609/1206] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:08:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:08:51.821882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c9aabc-5c60-4d15-8437-8dc7dfd4a916 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:51.822484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c9aabc-5c60-4d15-8437-8dc7dfd4a916 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:51.822484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c9aabc-5c60-4d15-8437-8dc7dfd4a916 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:51.845342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-70c9aabc-5c60-4d15-8437-8dc7dfd4a916 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:08:51.846510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-70c9aabc-5c60-4d15-8437-8dc7dfd4a916 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:08:51.922237 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 598/1207] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:08:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:08:52.221359 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0082edbc-c15b-43b4-ab1e-49e8a98e2a28 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:52.339599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0082edbc-c15b-43b4-ab1e-49e8a98e2a28 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:52.377939 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab2d83c-c069-466b-b793-264f0194a79e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:52.378649 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab2d83c-c069-466b-b793-264f0194a79e None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:52.378929 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab2d83c-c069-466b-b793-264f0194a79e None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:52.379248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab2d83c-c069-466b-b793-264f0194a79e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:52.391836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab2d83c-c069-466b-b793-264f0194a79e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:08:52.397378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-eab2d83c-c069-466b-b793-264f0194a79e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:08:52.397378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:08:52.397378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:08:52.397378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:08:52.397378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:08:52.397378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:08:52.397378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:08:52.397378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:08:52.397378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:08:52.397378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:08:52.397378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:08:52.397378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:08:52.397378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:08:52.397378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:08:52.397378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:08:52.397378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:08:52.398329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:08:52.398329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:08:52.398329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:08:52.398329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:08:52.398329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:08:52.398329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:08:52.398329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:08:52.398329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:08:52.398329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:08:52.398329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:08:52.398329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:08:52.398329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:08:52.398329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:08:52.398329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:08:52.398329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:08:52.398329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:08:52.399721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:08:52.399721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:08:52.399721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:08:52.399721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:08:52.399721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:08:52.399721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:08:52.399721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:08:52.399721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:08:52.399721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:08:52.399721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:08:52.399721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:08:52.399721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:08:52.406898 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 610/1208] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:08:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:08:52.407081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c9aabc-5c60-4d15-8437-8dc7dfd4a916 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:52.455513 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:52.527349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c9aabc-5c60-4d15-8437-8dc7dfd4a916 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:52.567346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046a1ba3-cdfc-4e6a-929a-ea9487466e7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:52.567963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046a1ba3-cdfc-4e6a-929a-ea9487466e7a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:52.568416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046a1ba3-cdfc-4e6a-929a-ea9487466e7a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:52.568859 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046a1ba3-cdfc-4e6a-929a-ea9487466e7a None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:52.569350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046a1ba3-cdfc-4e6a-929a-ea9487466e7a None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:08:52.569653 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046a1ba3-cdfc-4e6a-929a-ea9487466e7a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:52.571049 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046a1ba3-cdfc-4e6a-929a-ea9487466e7a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:52.579570 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 599/1209] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:08:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:08:52.585213 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab2d83c-c069-466b-b793-264f0194a79e None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:52.659095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:52.724986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab2d83c-c069-466b-b793-264f0194a79e None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:52.763006 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89e7973-a244-4b49-8976-61f9bd6735f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:52.763672 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89e7973-a244-4b49-8976-61f9bd6735f8 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:52.764028 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89e7973-a244-4b49-8976-61f9bd6735f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:52.764280 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89e7973-a244-4b49-8976-61f9bd6735f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:52.776957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89e7973-a244-4b49-8976-61f9bd6735f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:52.778680 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89e7973-a244-4b49-8976-61f9bd6735f8 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:52.795216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-e89e7973-a244-4b49-8976-61f9bd6735f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:08:52.795216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:08:52.795216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:08:52.795216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:08:52.795216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:08:52.795216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:08:52.795216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:08:52.795216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:08:52.795216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:08:52.795216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:08:52.795216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:08:52.795216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:08:52.795216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:08:52.795216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:08:52.795216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:08:52.795216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:08:52.796474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:08:52.796474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:08:52.796474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:08:52.796474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:08:52.796474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:08:52.796474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:08:52.796474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:08:52.796474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:08:52.796474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:08:52.796474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:08:52.796474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:08:52.796474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:08:52.796474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:08:52.796474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:08:52.796474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:08:52.796474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:08:52.796474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:08:52.797626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:08:52.797626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:08:52.797626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:08:52.797626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 611/1210] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:08:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:08:52.802753 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046a1ba3-cdfc-4e6a-929a-ea9487466e7a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:52.952677 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:52.959212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046a1ba3-cdfc-4e6a-929a-ea9487466e7a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:52.997163 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8377b53c-d4be-4c64-aaf8-c6d699c3d777 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:52.998007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8377b53c-d4be-4c64-aaf8-c6d699c3d777 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:52.998379 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8377b53c-d4be-4c64-aaf8-c6d699c3d777 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:52.998743 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8377b53c-d4be-4c64-aaf8-c6d699c3d777 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:52.999504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8377b53c-d4be-4c64-aaf8-c6d699c3d777 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:08:52.999893 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8377b53c-d4be-4c64-aaf8-c6d699c3d777 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:53.001344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8377b53c-d4be-4c64-aaf8-c6d699c3d777 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:53.013990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 600/1211] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:08:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:08:53.019774 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89e7973-a244-4b49-8976-61f9bd6735f8 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:53.147108 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89e7973-a244-4b49-8976-61f9bd6735f8 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:53.148872 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:53.196324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dbddc6-2da0-4df1-8ee8-13e3f7c1d9af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:53.197023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dbddc6-2da0-4df1-8ee8-13e3f7c1d9af None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:53.197380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dbddc6-2da0-4df1-8ee8-13e3f7c1d9af None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:53.197640 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dbddc6-2da0-4df1-8ee8-13e3f7c1d9af None admin] PATH_INFO: `/v3/users/placement` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:53.207513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dbddc6-2da0-4df1-8ee8-13e3f7c1d9af None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:53.209005 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dbddc6-2da0-4df1-8ee8-13e3f7c1d9af None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:53.222195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-04dbddc6-2da0-4df1-8ee8-13e3f7c1d9af None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 13 01:08:53.222195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:08:53.222195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:08:53.222195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:08:53.222195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:08:53.222195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:08:53.222195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:08:53.222195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:08:53.222195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:08:53.222195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:08:53.222195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:08:53.222195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:08:53.222195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:08:53.222195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:08:53.222195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:08:53.222195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:08:53.223496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:08:53.223496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:08:53.223496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:08:53.223496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:08:53.223496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:08:53.223496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:08:53.223496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:08:53.223496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:08:53.223496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:08:53.223496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:08:53.223496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:08:53.223496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:08:53.223496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:08:53.223496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:08:53.223496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:08:53.223496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:08:53.223496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:08:53.223496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:08:53.224803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:08:53.224803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:08:53.224803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:08:53.224803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:08:53.224803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:08:53.224803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:08:53.224803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:08:53.224803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:08:53.224803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 13 01:08:53.224803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:08:53.230148 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 612/1212] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:08:53 2022] GET /identity/v3/users/placement => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:08:53.230287 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8377b53c-d4be-4c64-aaf8-c6d699c3d777 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:53.349826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8377b53c-d4be-4c64-aaf8-c6d699c3d777 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:53.396611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7509b3c2-8101-406a-bec1-62d83f043c2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:53.397486 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7509b3c2-8101-406a-bec1-62d83f043c2b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:53.397801 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7509b3c2-8101-406a-bec1-62d83f043c2b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:53.398168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7509b3c2-8101-406a-bec1-62d83f043c2b None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:53.398918 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7509b3c2-8101-406a-bec1-62d83f043c2b None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:08:53.399230 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7509b3c2-8101-406a-bec1-62d83f043c2b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:53.400963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7509b3c2-8101-406a-bec1-62d83f043c2b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:53.431128 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 601/1213] 10.222.0.15 () {56 vars in 1143 bytes} [Tue Sep 13 01:08:53 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:08:53.437540 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dbddc6-2da0-4df1-8ee8-13e3f7c1d9af None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:53.552012 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dbddc6-2da0-4df1-8ee8-13e3f7c1d9af None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:53.593569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c17fae-f27d-4958-aed0-1f0094cd850f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:53.594188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c17fae-f27d-4958-aed0-1f0094cd850f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:53.594446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c17fae-f27d-4958-aed0-1f0094cd850f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:53.594736 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c17fae-f27d-4958-aed0-1f0094cd850f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:53.626798 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:53.673998 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c17fae-f27d-4958-aed0-1f0094cd850f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:53.675906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c17fae-f27d-4958-aed0-1f0094cd850f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:53.802805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 613/1214] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:08:53 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 01:08:53.809223 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7509b3c2-8101-406a-bec1-62d83f043c2b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:53.838278 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:53.942033 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7509b3c2-8101-406a-bec1-62d83f043c2b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:53.982973 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ad2fab-7fb2-4bfe-9177-7fe9e6d48095 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:53.983691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ad2fab-7fb2-4bfe-9177-7fe9e6d48095 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:53.983900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ad2fab-7fb2-4bfe-9177-7fe9e6d48095 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:53.984171 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ad2fab-7fb2-4bfe-9177-7fe9e6d48095 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:53.997311 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ad2fab-7fb2-4bfe-9177-7fe9e6d48095 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:53.999057 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ad2fab-7fb2-4bfe-9177-7fe9e6d48095 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:54.017068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-97ad2fab-7fb2-4bfe-9177-7fe9e6d48095 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:08:54.017068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:08:54.017068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:08:54.017068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:08:54.017068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:08:54.017068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:08:54.017068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:08:54.017068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:08:54.017068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:08:54.017068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:08:54.017068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:08:54.017068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:08:54.017068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:08:54.017068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:08:54.017068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:08:54.017068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:08:54.018306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:08:54.018306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:08:54.018306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:08:54.018306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:08:54.018306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:08:54.018306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:08:54.018306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:08:54.018306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:08:54.018306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:08:54.018306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:08:54.018306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:08:54.018306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:08:54.018306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:08:54.018306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:08:54.018306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:08:54.018306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:08:54.018306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:08:54.019512 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:08:54.019512 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:08:54.019512 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:08:54.019512 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 602/1215] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:08:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:08:54.024568 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c17fae-f27d-4958-aed0-1f0094cd850f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:54.102758 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:54.156471 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c17fae-f27d-4958-aed0-1f0094cd850f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:54.194444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b756a071-63ef-4c6d-90f8-097ca2c7e2fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:54.195118 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b756a071-63ef-4c6d-90f8-097ca2c7e2fb None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:54.195397 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b756a071-63ef-4c6d-90f8-097ca2c7e2fb None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:54.195736 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b756a071-63ef-4c6d-90f8-097ca2c7e2fb None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:54.196364 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b756a071-63ef-4c6d-90f8-097ca2c7e2fb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:08:54.196660 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b756a071-63ef-4c6d-90f8-097ca2c7e2fb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:54.197977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b756a071-63ef-4c6d-90f8-097ca2c7e2fb None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:54.209513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 614/1216] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:08:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:08:54.215484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ad2fab-7fb2-4bfe-9177-7fe9e6d48095 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:54.287918 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:54.324274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ad2fab-7fb2-4bfe-9177-7fe9e6d48095 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:54.365768 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87a9353-536b-45b8-b1dc-6296a718b081 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:54.366411 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87a9353-536b-45b8-b1dc-6296a718b081 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:54.366696 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87a9353-536b-45b8-b1dc-6296a718b081 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:54.367009 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87a9353-536b-45b8-b1dc-6296a718b081 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:54.374385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a87a9353-536b-45b8-b1dc-6296a718b081 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:54.375860 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a87a9353-536b-45b8-b1dc-6296a718b081 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:54.391583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-a87a9353-536b-45b8-b1dc-6296a718b081 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:08:54.391583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:08:54.391583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:08:54.391583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:08:54.391583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:08:54.391583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:08:54.391583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:08:54.391583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:08:54.391583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:08:54.391583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:08:54.391583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:08:54.391583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:08:54.391583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:08:54.391583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:08:54.391583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:08:54.392551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:08:54.392551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:08:54.392551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:08:54.392551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:08:54.392551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:08:54.392551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:08:54.392551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:08:54.392551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:08:54.392551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:08:54.392551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:08:54.392551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:08:54.392551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:08:54.392551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:08:54.392551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:08:54.392551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:08:54.392551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:08:54.392551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:08:54.393469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:08:54.393469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:08:54.393469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:08:54.393469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:08:54.393469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:08:54.393469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:08:54.393469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:08:54.393469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:08:54.393469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 603/1217] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:08:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:08:54.400431 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b756a071-63ef-4c6d-90f8-097ca2c7e2fb None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:54.524005 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b756a071-63ef-4c6d-90f8-097ca2c7e2fb None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:54.549806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:54.563453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1955e456-116e-40f7-ac55-9fc2f9708a05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:54.564122 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1955e456-116e-40f7-ac55-9fc2f9708a05 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:54.564420 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1955e456-116e-40f7-ac55-9fc2f9708a05 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:54.564711 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1955e456-116e-40f7-ac55-9fc2f9708a05 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:54.565403 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1955e456-116e-40f7-ac55-9fc2f9708a05 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:08:54.565717 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1955e456-116e-40f7-ac55-9fc2f9708a05 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:54.567218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1955e456-116e-40f7-ac55-9fc2f9708a05 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:54.582406 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 615/1218] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:08:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:08:54.590401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87a9353-536b-45b8-b1dc-6296a718b081 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:54.704416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87a9353-536b-45b8-b1dc-6296a718b081 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:54.733088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:54.739718 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61d199f-0ada-4463-b7df-705e70922fe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:54.740335 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61d199f-0ada-4463-b7df-705e70922fe6 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:54.740620 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61d199f-0ada-4463-b7df-705e70922fe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:54.740885 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61d199f-0ada-4463-b7df-705e70922fe6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:54.741637 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61d199f-0ada-4463-b7df-705e70922fe6 None admin] RBAC: Adding query filter params (role.id=71b1e7c4b5af44ddbb7a517760ad0119, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=8eaff1b843a145a49994dcf089f44315) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:08:54.741940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61d199f-0ada-4463-b7df-705e70922fe6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:54.743383 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61d199f-0ada-4463-b7df-705e70922fe6 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:54.749510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 604/1219] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:08:54 2022] GET /identity/v3/role_assignments?user.id=8eaff1b843a145a49994dcf089f44315&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=71b1e7c4b5af44ddbb7a517760ad0119 => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:08:56.874179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef355da-c5e8-4e45-b470-c94ebf0cfe2c None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:56.874769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef355da-c5e8-4e45-b470-c94ebf0cfe2c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:56.874769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef355da-c5e8-4e45-b470-c94ebf0cfe2c None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:56.876015 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 616/1220] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:08:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:08:56.881449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296ab11f-d40a-4ed7-ab3d-9e05b280a758 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:56.881992 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296ab11f-d40a-4ed7-ab3d-9e05b280a758 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:56.882192 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296ab11f-d40a-4ed7-ab3d-9e05b280a758 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:56.902862 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-296ab11f-d40a-4ed7-ab3d-9e05b280a758 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:08:56.904769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-296ab11f-d40a-4ed7-ab3d-9e05b280a758 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:08:56.979866 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 605/1221] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:08:56 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:08:57.251387 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef355da-c5e8-4e45-b470-c94ebf0cfe2c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:57.361843 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef355da-c5e8-4e45-b470-c94ebf0cfe2c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:57.399493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219e42dd-90ac-409d-a176-cb834211c540 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:57.400156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219e42dd-90ac-409d-a176-cb834211c540 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:57.400457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219e42dd-90ac-409d-a176-cb834211c540 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:57.400708 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219e42dd-90ac-409d-a176-cb834211c540 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:57.412678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219e42dd-90ac-409d-a176-cb834211c540 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:08:57.417453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-219e42dd-90ac-409d-a176-cb834211c540 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:08:57.417453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:08:57.417453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:08:57.417453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:08:57.417453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:08:57.417453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:08:57.417453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:08:57.417453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:08:57.417453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:08:57.417453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:08:57.417453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:08:57.417453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:08:57.417453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:08:57.417453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:08:57.417453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:08:57.417453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:08:57.418616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:08:57.418616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:08:57.418616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:08:57.418616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:08:57.418616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:08:57.418616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:08:57.418616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:08:57.418616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:08:57.418616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:08:57.418616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:08:57.418616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:08:57.418616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:08:57.418616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:08:57.418616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:08:57.418616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:08:57.418616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:08:57.419776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:08:57.419776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:08:57.419776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:08:57.419776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:08:57.419776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:08:57.419776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:08:57.419776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:08:57.419776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:08:57.419776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:08:57.419776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:08:57.419776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:08:57.419776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:08:57.423012 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 617/1222] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:08:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:08:57.424748 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296ab11f-d40a-4ed7-ab3d-9e05b280a758 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:57.543089 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296ab11f-d40a-4ed7-ab3d-9e05b280a758 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:57.570369 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:57.581548 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38470b97-3c32-464b-bdd6-2fd295252880 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:57.582182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38470b97-3c32-464b-bdd6-2fd295252880 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:57.582454 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38470b97-3c32-464b-bdd6-2fd295252880 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:57.582710 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38470b97-3c32-464b-bdd6-2fd295252880 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:57.583399 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38470b97-3c32-464b-bdd6-2fd295252880 None admin] RBAC: Adding query filter params (name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:08:57.583702 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38470b97-3c32-464b-bdd6-2fd295252880 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:57.585047 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38470b97-3c32-464b-bdd6-2fd295252880 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:57.593152 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 606/1223] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:08:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:08:57.599510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219e42dd-90ac-409d-a176-cb834211c540 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:57.650960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:57.714330 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219e42dd-90ac-409d-a176-cb834211c540 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:57.752749 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9630a453-fc5d-4bec-a06c-6ffbd0bca3a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:57.753501 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9630a453-fc5d-4bec-a06c-6ffbd0bca3a0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:57.753788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9630a453-fc5d-4bec-a06c-6ffbd0bca3a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:57.754071 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9630a453-fc5d-4bec-a06c-6ffbd0bca3a0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:57.765778 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9630a453-fc5d-4bec-a06c-6ffbd0bca3a0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:57.767285 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9630a453-fc5d-4bec-a06c-6ffbd0bca3a0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:57.784062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-9630a453-fc5d-4bec-a06c-6ffbd0bca3a0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:08:57.784062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:08:57.784062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:08:57.784062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:08:57.784062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:08:57.784062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:08:57.784062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:08:57.784062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:08:57.784062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:08:57.784062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:08:57.784062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:08:57.784062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:08:57.784062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:08:57.784062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:08:57.784062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:08:57.784062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:08:57.785132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:08:57.785132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:08:57.785132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:08:57.785132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:08:57.785132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:08:57.785132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:08:57.785132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:08:57.785132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:08:57.785132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:08:57.785132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:08:57.785132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:08:57.785132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:08:57.785132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:08:57.785132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:08:57.785132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:08:57.785132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:08:57.785132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:08:57.786044 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:08:57.786044 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:08:57.786044 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:08:57.786044 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 618/1224] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:08:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:08:57.791901 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38470b97-3c32-464b-bdd6-2fd295252880 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:57.894384 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38470b97-3c32-464b-bdd6-2fd295252880 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:57.928332 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a33597-ad15-4723-a206-b6407289c833 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:57.928941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a33597-ad15-4723-a206-b6407289c833 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:57.929221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a33597-ad15-4723-a206-b6407289c833 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:57.929494 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a33597-ad15-4723-a206-b6407289c833 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:57.930127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a33597-ad15-4723-a206-b6407289c833 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:08:57.930428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a33597-ad15-4723-a206-b6407289c833 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:57.931818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a33597-ad15-4723-a206-b6407289c833 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:57.944974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 607/1225] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:08:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:08:57.949903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9630a453-fc5d-4bec-a06c-6ffbd0bca3a0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:58.057931 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:58.066918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9630a453-fc5d-4bec-a06c-6ffbd0bca3a0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:58.109432 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a8d578-d965-414f-b9b6-6dbea80e6a6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:58.110065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a8d578-d965-414f-b9b6-6dbea80e6a6a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:58.110343 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a8d578-d965-414f-b9b6-6dbea80e6a6a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:58.110615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a8d578-d965-414f-b9b6-6dbea80e6a6a None admin] PATH_INFO: `/v3/users/placement` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:58.119599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a8d578-d965-414f-b9b6-6dbea80e6a6a None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:58.121368 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a8d578-d965-414f-b9b6-6dbea80e6a6a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:58.134783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-79a8d578-d965-414f-b9b6-6dbea80e6a6a None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 13 01:08:58.134783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:08:58.134783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:08:58.134783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:08:58.134783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:08:58.134783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:08:58.134783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:08:58.134783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:08:58.134783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:08:58.134783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:08:58.134783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:08:58.134783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:08:58.134783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:08:58.134783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:08:58.134783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:08:58.134783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:08:58.135946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:08:58.135946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:08:58.135946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:08:58.135946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:08:58.135946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:08:58.135946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:08:58.135946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:08:58.135946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:08:58.135946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:08:58.135946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:08:58.135946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:08:58.135946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:08:58.135946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:08:58.135946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:08:58.135946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:08:58.135946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:08:58.135946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:08:58.135946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:08:58.137074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:08:58.137074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:08:58.137074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:08:58.137074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:08:58.137074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:08:58.137074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:08:58.137074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:08:58.137074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:08:58.137074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 13 01:08:58.137074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:08:58.137074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 619/1226] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:08:57 2022] GET /identity/v3/users/placement => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:08:58.143067 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a33597-ad15-4723-a206-b6407289c833 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:58.190548 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:58.246738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a33597-ad15-4723-a206-b6407289c833 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:58.283330 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b7f8c0-d41a-49bb-90b5-76caea716786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:58.283945 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b7f8c0-d41a-49bb-90b5-76caea716786 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:58.284229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b7f8c0-d41a-49bb-90b5-76caea716786 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:58.284498 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b7f8c0-d41a-49bb-90b5-76caea716786 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:58.285215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b7f8c0-d41a-49bb-90b5-76caea716786 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:08:58.285536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b7f8c0-d41a-49bb-90b5-76caea716786 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:58.286975 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b7f8c0-d41a-49bb-90b5-76caea716786 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:58.306316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 608/1227] 10.222.0.15 () {56 vars in 1143 bytes} [Tue Sep 13 01:08:58 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:08:58.312291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a8d578-d965-414f-b9b6-6dbea80e6a6a None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:58.429164 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a8d578-d965-414f-b9b6-6dbea80e6a6a None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:58.463424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d992ce9-c20a-4ebc-84e9-ad26eef47634 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:58.465194 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d992ce9-c20a-4ebc-84e9-ad26eef47634 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:58.465466 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d992ce9-c20a-4ebc-84e9-ad26eef47634 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:58.465735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d992ce9-c20a-4ebc-84e9-ad26eef47634 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:58.476777 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d992ce9-c20a-4ebc-84e9-ad26eef47634 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:58.478208 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d992ce9-c20a-4ebc-84e9-ad26eef47634 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:58.495084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-0d992ce9-c20a-4ebc-84e9-ad26eef47634 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:08:58.495084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:08:58.495084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:08:58.495084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:08:58.495084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:08:58.495084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:08:58.495084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:08:58.495084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:08:58.495084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:08:58.495084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:08:58.495084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:08:58.495084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:08:58.495084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:08:58.495084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:08:58.495084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:08:58.495084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:08:58.496249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:08:58.496249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:08:58.496249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:08:58.496249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:08:58.496249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:08:58.496249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:08:58.496249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:08:58.496249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:08:58.496249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:08:58.496249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:08:58.496249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:08:58.496249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:08:58.496249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:08:58.496249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:08:58.496249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:08:58.496249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:08:58.496249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:08:58.497255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:08:58.497255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:08:58.497255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:08:58.497255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 620/1228] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:08:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:08:58.501950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b7f8c0-d41a-49bb-90b5-76caea716786 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:58.607355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b7f8c0-d41a-49bb-90b5-76caea716786 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:58.609660 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:58.645430 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693ecd39-6896-4894-bd1e-779c50682851 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:58.646049 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693ecd39-6896-4894-bd1e-779c50682851 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:58.646311 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693ecd39-6896-4894-bd1e-779c50682851 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:58.646597 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693ecd39-6896-4894-bd1e-779c50682851 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:58.647241 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693ecd39-6896-4894-bd1e-779c50682851 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:08:58.649631 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693ecd39-6896-4894-bd1e-779c50682851 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:58.649631 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693ecd39-6896-4894-bd1e-779c50682851 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:58.661274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 609/1229] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:08:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:08:58.667283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d992ce9-c20a-4ebc-84e9-ad26eef47634 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:58.676890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:58.780368 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d992ce9-c20a-4ebc-84e9-ad26eef47634 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:58.813998 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733395e2-1f53-405a-9e45-4f6e48196fa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:58.814646 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733395e2-1f53-405a-9e45-4f6e48196fa0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:58.814956 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733395e2-1f53-405a-9e45-4f6e48196fa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:58.815228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733395e2-1f53-405a-9e45-4f6e48196fa0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:58.821625 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733395e2-1f53-405a-9e45-4f6e48196fa0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:58.823083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733395e2-1f53-405a-9e45-4f6e48196fa0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:58.835903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-733395e2-1f53-405a-9e45-4f6e48196fa0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:08:58.835903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:08:58.835903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:08:58.835903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:08:58.835903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:08:58.835903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:08:58.835903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:08:58.835903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:08:58.835903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:08:58.835903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:08:58.835903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:08:58.835903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:08:58.835903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:08:58.835903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:08:58.835903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:08:58.837011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:08:58.837011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:08:58.837011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:08:58.837011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:08:58.837011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:08:58.837011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:08:58.837011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:08:58.837011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:08:58.837011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:08:58.837011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:08:58.837011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:08:58.837011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:08:58.837011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:08:58.837011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:08:58.837011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:08:58.837011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:08:58.837011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:08:58.838034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:08:58.838034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:08:58.838034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:08:58.838034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:08:58.838034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:08:58.838034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:08:58.838034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:08:58.838034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:08:58.838034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 621/1230] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:08:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:08:58.843553 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693ecd39-6896-4894-bd1e-779c50682851 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:58.955630 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693ecd39-6896-4894-bd1e-779c50682851 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:58.992141 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43094029-9275-4ac8-8fdc-8f130fb2b36b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:58.992748 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43094029-9275-4ac8-8fdc-8f130fb2b36b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:58.993027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43094029-9275-4ac8-8fdc-8f130fb2b36b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:58.993297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43094029-9275-4ac8-8fdc-8f130fb2b36b None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:58.993976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43094029-9275-4ac8-8fdc-8f130fb2b36b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:08:58.994280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43094029-9275-4ac8-8fdc-8f130fb2b36b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:58.995750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43094029-9275-4ac8-8fdc-8f130fb2b36b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:59.008713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 610/1231] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:08:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:08:59.015906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733395e2-1f53-405a-9e45-4f6e48196fa0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:08:59.096536 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:08:59.131701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733395e2-1f53-405a-9e45-4f6e48196fa0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:08:59.168000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d73fba-4d0f-4d65-8e45-99b299c3d6fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:08:59.169040 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d73fba-4d0f-4d65-8e45-99b299c3d6fb None admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:08:59.169345 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d73fba-4d0f-4d65-8e45-99b299c3d6fb None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:08:59.169659 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d73fba-4d0f-4d65-8e45-99b299c3d6fb None admin] PATH_INFO: `/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/8eaff1b843a145a49994dcf089f44315/roles/71b1e7c4b5af44ddbb7a517760ad0119` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:08:59.188363 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d73fba-4d0f-4d65-8e45-99b299c3d6fb None admin] RBAC: Authorizing `identity:create_grant(project_id=e5c1519fdae742f4a25eb7cd5476a8cb, user_id=8eaff1b843a145a49994dcf089f44315, role_id=71b1e7c4b5af44ddbb7a517760ad0119)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:08:59.189831 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d73fba-4d0f-4d65-8e45-99b299c3d6fb None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:08:59.205805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 622/1232] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:08:59 2022] PUT /identity/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/8eaff1b843a145a49994dcf089f44315/roles/71b1e7c4b5af44ddbb7a517760ad0119 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:09:01.863989 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970a2240-b347-49e7-b166-094c542e2c81 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:01.864543 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970a2240-b347-49e7-b166-094c542e2c81 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:01.864632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970a2240-b347-49e7-b166-094c542e2c81 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:01.866515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 611/1233] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:09:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:09:01.871828 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7077fa-9f16-44a5-aa02-ed684fe595f6 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:01.872720 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7077fa-9f16-44a5-aa02-ed684fe595f6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:01.872897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7077fa-9f16-44a5-aa02-ed684fe595f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:01.891746 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5f7077fa-9f16-44a5-aa02-ed684fe595f6 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:09:01.892919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5f7077fa-9f16-44a5-aa02-ed684fe595f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:09:01.960440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 623/1234] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:09:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:09:02.322215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970a2240-b347-49e7-b166-094c542e2c81 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:02.352891 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:02.449059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970a2240-b347-49e7-b166-094c542e2c81 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:02.485059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357c0dfa-8562-47ec-8890-5854ff42b4a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:02.485821 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357c0dfa-8562-47ec-8890-5854ff42b4a4 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:02.485999 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357c0dfa-8562-47ec-8890-5854ff42b4a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:02.486221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357c0dfa-8562-47ec-8890-5854ff42b4a4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:02.499185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357c0dfa-8562-47ec-8890-5854ff42b4a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:09:02.503914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-357c0dfa-8562-47ec-8890-5854ff42b4a4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:09:02.503914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:02.503914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:02.503914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:02.503914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:02.503914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:02.503914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:02.503914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:02.503914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:02.503914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:02.503914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:02.503914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:02.503914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:09:02.503914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:09:02.503914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:09:02.503914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:09:02.505043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:09:02.505043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:09:02.505043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:09:02.505043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:09:02.505043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:02.505043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:02.505043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:02.505043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:02.505043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:02.505043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:02.505043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:02.505043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:02.505043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:02.505043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:02.505043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:02.505043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:02.506029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:02.506029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:02.506029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:02.506029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:02.506029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:09:02.506029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:09:02.506029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:09:02.506029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:09:02.506029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:09:02.506029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:09:02.506029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 01:09:02.506029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:09:02.511012 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 612/1235] 10.222.0.15 () {56 vars in 1102 bytes} [Tue Sep 13 01:09:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:02.512014 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7077fa-9f16-44a5-aa02-ed684fe595f6 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:02.565293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:02.623229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7077fa-9f16-44a5-aa02-ed684fe595f6 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:02.663618 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c84647-b5af-488b-834f-ea9fdd71d1e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:02.664225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c84647-b5af-488b-834f-ea9fdd71d1e5 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:02.664501 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c84647-b5af-488b-834f-ea9fdd71d1e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:02.664770 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c84647-b5af-488b-834f-ea9fdd71d1e5 None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:02.665419 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c84647-b5af-488b-834f-ea9fdd71d1e5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:02.665717 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c84647-b5af-488b-834f-ea9fdd71d1e5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:02.667097 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c84647-b5af-488b-834f-ea9fdd71d1e5 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:02.675492 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 624/1236] 10.222.0.15 () {56 vars in 1103 bytes} [Tue Sep 13 01:09:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:02.681464 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357c0dfa-8562-47ec-8890-5854ff42b4a4 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:02.788703 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:02.806782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357c0dfa-8562-47ec-8890-5854ff42b4a4 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:02.844698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b00468-843c-4c4c-ad19-d3428c8d7f81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:02.845344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b00468-843c-4c4c-ad19-d3428c8d7f81 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:02.845610 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b00468-843c-4c4c-ad19-d3428c8d7f81 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:02.845880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b00468-843c-4c4c-ad19-d3428c8d7f81 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:02.857226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b00468-843c-4c4c-ad19-d3428c8d7f81 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:02.858744 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b00468-843c-4c4c-ad19-d3428c8d7f81 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:02.874928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-c8b00468-843c-4c4c-ad19-d3428c8d7f81 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:09:02.874928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:02.874928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:02.874928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:02.874928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:02.874928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:02.874928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:02.874928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:02.874928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:02.874928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:02.874928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:02.874928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:02.874928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:09:02.874928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:09:02.874928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:09:02.874928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:09:02.876351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:02.876351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:02.876351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:02.876351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:02.876351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:02.876351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:02.876351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:02.876351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:02.876351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:02.876351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:02.876351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:02.876351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:02.876351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:02.876351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:02.876351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:02.876351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:02.876351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:09:02.877225 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:09:02.877225 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:09:02.877225 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:09:02.877225 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 613/1237] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:09:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:02.882071 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c84647-b5af-488b-834f-ea9fdd71d1e5 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:02.994782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c84647-b5af-488b-834f-ea9fdd71d1e5 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:03.005340 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:03.033038 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d16b278-ded8-42a0-8935-497f03d69fb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:03.033638 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d16b278-ded8-42a0-8935-497f03d69fb0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:03.033911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d16b278-ded8-42a0-8935-497f03d69fb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:03.034202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d16b278-ded8-42a0-8935-497f03d69fb0 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:03.034961 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d16b278-ded8-42a0-8935-497f03d69fb0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:03.035304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d16b278-ded8-42a0-8935-497f03d69fb0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:03.036789 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d16b278-ded8-42a0-8935-497f03d69fb0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:03.050443 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 625/1238] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:09:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:03.058105 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b00468-843c-4c4c-ad19-d3428c8d7f81 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:03.180336 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b00468-843c-4c4c-ad19-d3428c8d7f81 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:03.219669 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f46854d-6a0e-4d88-a68a-072f2b7d4c3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:03.220514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f46854d-6a0e-4d88-a68a-072f2b7d4c3c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:03.220966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f46854d-6a0e-4d88-a68a-072f2b7d4c3c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:03.221242 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f46854d-6a0e-4d88-a68a-072f2b7d4c3c None admin] PATH_INFO: `/v3/users/placement` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:03.224285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:03.230705 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f46854d-6a0e-4d88-a68a-072f2b7d4c3c None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:03.233205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f46854d-6a0e-4d88-a68a-072f2b7d4c3c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:03.246883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-4f46854d-6a0e-4d88-a68a-072f2b7d4c3c None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 13 01:09:03.246883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:03.246883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:03.246883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:03.246883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:03.246883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:03.246883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:03.246883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:03.246883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:03.246883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:03.246883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:03.246883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:03.246883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:09:03.246883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:09:03.246883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:09:03.246883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:09:03.248088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:03.248088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:03.248088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:09:03.248088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:09:03.248088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:09:03.248088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:09:03.248088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:03.248088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:03.248088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:03.248088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:03.248088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:03.248088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:03.248088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:03.248088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:03.248088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:03.248088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:03.248088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:03.248088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:03.249617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:03.249617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:03.249617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:09:03.249617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:09:03.249617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:09:03.249617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:09:03.249617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:09:03.249617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:09:03.249617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 13 01:09:03.249617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:09:03.249617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 614/1239] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:09:03 2022] GET /identity/v3/users/placement => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:03.255279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d16b278-ded8-42a0-8935-497f03d69fb0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:03.364001 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d16b278-ded8-42a0-8935-497f03d69fb0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:03.401259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc03e78-45cb-43f2-b3b6-1ea0f9c41b7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:03.401856 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc03e78-45cb-43f2-b3b6-1ea0f9c41b7c None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:03.402165 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc03e78-45cb-43f2-b3b6-1ea0f9c41b7c None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:03.402478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc03e78-45cb-43f2-b3b6-1ea0f9c41b7c None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:03.403337 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc03e78-45cb-43f2-b3b6-1ea0f9c41b7c None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:03.403693 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc03e78-45cb-43f2-b3b6-1ea0f9c41b7c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:03.405247 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc03e78-45cb-43f2-b3b6-1ea0f9c41b7c None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:03.425374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 626/1240] 10.222.0.15 () {56 vars in 1143 bytes} [Tue Sep 13 01:09:03 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:03.431026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f46854d-6a0e-4d88-a68a-072f2b7d4c3c None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:03.547101 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f46854d-6a0e-4d88-a68a-072f2b7d4c3c None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:03.587000 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cb99f5-fabe-48d3-8f63-8654bbc20694 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:03.587752 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cb99f5-fabe-48d3-8f63-8654bbc20694 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:03.588108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cb99f5-fabe-48d3-8f63-8654bbc20694 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:03.588514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cb99f5-fabe-48d3-8f63-8654bbc20694 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:03.659633 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:03.666663 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28cb99f5-fabe-48d3-8f63-8654bbc20694 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:03.668178 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28cb99f5-fabe-48d3-8f63-8654bbc20694 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:03.781658 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 615/1241] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:09:03 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 01:09:03.788010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc03e78-45cb-43f2-b3b6-1ea0f9c41b7c None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:03.819834 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:03.902682 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc03e78-45cb-43f2-b3b6-1ea0f9c41b7c None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:03.942277 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799601b3-93c4-417e-a838-7f5a30a877a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:03.943099 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799601b3-93c4-417e-a838-7f5a30a877a9 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:03.943378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799601b3-93c4-417e-a838-7f5a30a877a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:03.943750 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799601b3-93c4-417e-a838-7f5a30a877a9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:03.954811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799601b3-93c4-417e-a838-7f5a30a877a9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:03.956697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799601b3-93c4-417e-a838-7f5a30a877a9 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:03.975326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-799601b3-93c4-417e-a838-7f5a30a877a9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:09:03.975326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:03.975326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:03.975326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:03.975326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:03.975326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:03.975326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:03.975326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:03.975326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:03.975326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:03.975326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:03.975326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:03.975326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:09:03.975326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:09:03.975326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:09:03.975326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:09:03.976327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:03.976327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:03.976327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:03.976327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:03.976327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:03.976327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:03.976327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:03.976327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:03.976327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:03.976327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:03.976327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:03.976327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:03.976327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:03.976327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:03.976327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:03.976327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:03.976327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:09:03.977252 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:09:03.977252 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:09:03.977252 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:09:03.977252 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 627/1242] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:09:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 01:09:03.983345 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cb99f5-fabe-48d3-8f63-8654bbc20694 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:04.084295 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:04.091883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cb99f5-fabe-48d3-8f63-8654bbc20694 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:04.135697 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9003e20c-a40c-47ca-8f60-5c67ec573d69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:04.136496 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9003e20c-a40c-47ca-8f60-5c67ec573d69 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:04.136780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9003e20c-a40c-47ca-8f60-5c67ec573d69 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:04.137051 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9003e20c-a40c-47ca-8f60-5c67ec573d69 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:04.138402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9003e20c-a40c-47ca-8f60-5c67ec573d69 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:04.138812 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9003e20c-a40c-47ca-8f60-5c67ec573d69 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:04.140549 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9003e20c-a40c-47ca-8f60-5c67ec573d69 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:04.154412 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 616/1243] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:09:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:04.161000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799601b3-93c4-417e-a838-7f5a30a877a9 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:04.259007 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:04.282542 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799601b3-93c4-417e-a838-7f5a30a877a9 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:04.321893 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0fb99a-819c-4410-8169-e8083ee5b062 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:04.322551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0fb99a-819c-4410-8169-e8083ee5b062 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:04.322818 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0fb99a-819c-4410-8169-e8083ee5b062 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:04.323121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0fb99a-819c-4410-8169-e8083ee5b062 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:04.330046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0fb99a-819c-4410-8169-e8083ee5b062 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:04.331532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0fb99a-819c-4410-8169-e8083ee5b062 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:04.343422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-2d0fb99a-819c-4410-8169-e8083ee5b062 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:09:04.343422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:04.343422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:04.343422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:04.343422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:04.343422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:04.343422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:04.343422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:04.343422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:04.343422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:04.343422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:04.343422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:04.343422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:09:04.343422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:09:04.343422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:09:04.344549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:09:04.344549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:04.344549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:04.344549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:04.344549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:04.344549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:04.344549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:04.344549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:04.344549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:04.344549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:04.344549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:04.344549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:04.344549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:04.344549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:04.344549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:04.344549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:04.344549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:04.345469 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:09:04.345469 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:09:04.345469 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:09:04.345469 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:09:04.345469 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:09:04.345469 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:09:04.345469 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:09:04.345469 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:09:04.345469 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 628/1244] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:09:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:04.350610 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9003e20c-a40c-47ca-8f60-5c67ec573d69 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:04.465299 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9003e20c-a40c-47ca-8f60-5c67ec573d69 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:04.507899 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32891837-cb5b-4307-b39e-837c433b5565 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:04.508638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32891837-cb5b-4307-b39e-837c433b5565 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:04.509108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32891837-cb5b-4307-b39e-837c433b5565 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:04.509542 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32891837-cb5b-4307-b39e-837c433b5565 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:04.510425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32891837-cb5b-4307-b39e-837c433b5565 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:04.510817 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32891837-cb5b-4307-b39e-837c433b5565 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:04.514684 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32891837-cb5b-4307-b39e-837c433b5565 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:04.536809 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 617/1245] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:09:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:04.538578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0fb99a-819c-4410-8169-e8083ee5b062 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:04.659405 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0fb99a-819c-4410-8169-e8083ee5b062 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:04.696680 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbb0e77-422d-478f-bb92-4d795e3108b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:04.697436 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbb0e77-422d-478f-bb92-4d795e3108b2 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:04.697824 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbb0e77-422d-478f-bb92-4d795e3108b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:04.698191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbb0e77-422d-478f-bb92-4d795e3108b2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:04.699093 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbb0e77-422d-478f-bb92-4d795e3108b2 None admin] RBAC: Adding query filter params (role.id=71b1e7c4b5af44ddbb7a517760ad0119, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=8eaff1b843a145a49994dcf089f44315) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:04.699502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbb0e77-422d-478f-bb92-4d795e3108b2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:04.701542 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbb0e77-422d-478f-bb92-4d795e3108b2 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:04.718881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 629/1246] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:09:04 2022] GET /identity/v3/role_assignments?user.id=8eaff1b843a145a49994dcf089f44315&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=71b1e7c4b5af44ddbb7a517760ad0119 => generated 608 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:06.975722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fb896d-db4c-428c-afa4-cb1fc9adae6f None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:06.976285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fb896d-db4c-428c-afa4-cb1fc9adae6f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:06.976535 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fb896d-db4c-428c-afa4-cb1fc9adae6f None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:06.978092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 618/1247] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:09:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:09:06.983497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4414b64e-1fc8-4a9d-b87c-330c86587cd1 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:06.984056 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4414b64e-1fc8-4a9d-b87c-330c86587cd1 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:06.984056 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4414b64e-1fc8-4a9d-b87c-330c86587cd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:06.990288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:07.003820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4414b64e-1fc8-4a9d-b87c-330c86587cd1 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:09:07.004943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4414b64e-1fc8-4a9d-b87c-330c86587cd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:09:07.077961 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 630/1248] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:09:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:09:07.345577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fb896d-db4c-428c-afa4-cb1fc9adae6f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:07.365197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:07.460958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fb896d-db4c-428c-afa4-cb1fc9adae6f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:07.497223 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27524bb4-81b0-432c-a520-65ad7dfc8d74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:07.497862 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27524bb4-81b0-432c-a520-65ad7dfc8d74 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:07.498141 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27524bb4-81b0-432c-a520-65ad7dfc8d74 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:07.498411 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27524bb4-81b0-432c-a520-65ad7dfc8d74 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:07.508343 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27524bb4-81b0-432c-a520-65ad7dfc8d74 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:09:07.513148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-27524bb4-81b0-432c-a520-65ad7dfc8d74 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:09:07.513148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:07.513148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:07.513148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:07.513148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:07.513148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:07.513148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:07.513148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:07.513148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:07.513148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:07.513148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:07.513148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:07.513148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:09:07.513148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:09:07.513148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:09:07.513148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:09:07.514355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:09:07.514355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:09:07.514355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:09:07.514355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:09:07.514355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:07.514355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:07.514355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:07.514355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:07.514355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:07.514355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:07.514355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:07.514355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:07.514355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:07.514355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:07.514355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:07.514355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:07.515266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:07.515266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:07.515266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:07.515266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:07.515266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:09:07.515266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:09:07.515266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:09:07.515266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:09:07.515266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:09:07.515266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:09:07.515266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:09:07.515266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:09:07.515266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 619/1249] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:09:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:07.520452 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4414b64e-1fc8-4a9d-b87c-330c86587cd1 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:07.619837 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4414b64e-1fc8-4a9d-b87c-330c86587cd1 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:07.648914 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:07.655088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6e0432-d70c-4a5e-bde2-0b4c24ccd53f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:07.655701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6e0432-d70c-4a5e-bde2-0b4c24ccd53f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:07.655974 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6e0432-d70c-4a5e-bde2-0b4c24ccd53f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:07.656245 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6e0432-d70c-4a5e-bde2-0b4c24ccd53f None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:07.656903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6e0432-d70c-4a5e-bde2-0b4c24ccd53f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:07.657209 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6e0432-d70c-4a5e-bde2-0b4c24ccd53f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:07.658546 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6e0432-d70c-4a5e-bde2-0b4c24ccd53f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:07.666951 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 631/1250] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:09:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:07.673007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27524bb4-81b0-432c-a520-65ad7dfc8d74 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:07.754000 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:07.777533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27524bb4-81b0-432c-a520-65ad7dfc8d74 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:07.810274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d21d912-40a5-4d9f-b831-beaef568f9b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:07.810905 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d21d912-40a5-4d9f-b831-beaef568f9b5 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:07.811191 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d21d912-40a5-4d9f-b831-beaef568f9b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:07.811456 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d21d912-40a5-4d9f-b831-beaef568f9b5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:07.821059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d21d912-40a5-4d9f-b831-beaef568f9b5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:07.822480 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d21d912-40a5-4d9f-b831-beaef568f9b5 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:07.836958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-0d21d912-40a5-4d9f-b831-beaef568f9b5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:09:07.836958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:07.836958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:07.836958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:07.836958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:07.836958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:07.836958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:07.836958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:07.836958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:07.836958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:07.836958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:07.836958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:07.836958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:09:07.836958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:09:07.836958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:09:07.836958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:09:07.838125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:07.838125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:07.838125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:07.838125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:07.838125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:07.838125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:07.838125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:07.838125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:07.838125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:07.838125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:07.838125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:07.838125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:07.838125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:07.838125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:07.838125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:07.838125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:07.838125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:09:07.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:09:07.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:09:07.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:09:07.839321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 620/1251] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:09:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:07.843913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6e0432-d70c-4a5e-bde2-0b4c24ccd53f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:07.940424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6e0432-d70c-4a5e-bde2-0b4c24ccd53f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:07.972711 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb83c1b-8eab-465c-a628-4ed4f96335db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:07.973293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb83c1b-8eab-465c-a628-4ed4f96335db None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:07.973567 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb83c1b-8eab-465c-a628-4ed4f96335db None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:07.973833 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb83c1b-8eab-465c-a628-4ed4f96335db None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:07.974451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb83c1b-8eab-465c-a628-4ed4f96335db None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:07.974750 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb83c1b-8eab-465c-a628-4ed4f96335db None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:07.976111 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb83c1b-8eab-465c-a628-4ed4f96335db None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:07.987355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 632/1252] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:09:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:07.992839 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d21d912-40a5-4d9f-b831-beaef568f9b5 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:08.101141 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d21d912-40a5-4d9f-b831-beaef568f9b5 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:08.131634 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:08.140160 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811dfd3b-2a55-412a-8e73-0440ef95125a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:08.140780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811dfd3b-2a55-412a-8e73-0440ef95125a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:08.141057 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811dfd3b-2a55-412a-8e73-0440ef95125a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:08.141328 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811dfd3b-2a55-412a-8e73-0440ef95125a None admin] PATH_INFO: `/v3/users/placement` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:08.148505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811dfd3b-2a55-412a-8e73-0440ef95125a None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:08.149926 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811dfd3b-2a55-412a-8e73-0440ef95125a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:08.161349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-811dfd3b-2a55-412a-8e73-0440ef95125a None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 13 01:09:08.161349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:08.161349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:08.161349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:08.161349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:08.161349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:08.161349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:08.161349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:08.161349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:08.161349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:08.161349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:08.161349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:08.161349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:09:08.161349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:09:08.161349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:09:08.161349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:09:08.162515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:08.162515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:08.162515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:09:08.162515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:09:08.162515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:09:08.162515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:09:08.162515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:08.162515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:08.162515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:08.162515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:08.162515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:08.162515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:08.162515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:08.162515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:08.162515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:08.162515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:08.162515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:08.162515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:08.163704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:08.163704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:08.163704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:09:08.163704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:09:08.163704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:09:08.163704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:09:08.163704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:09:08.163704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:09:08.163704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 13 01:09:08.163704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:09:08.163704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 621/1253] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:09:07 2022] GET /identity/v3/users/placement => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:08.168386 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb83c1b-8eab-465c-a628-4ed4f96335db None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:08.222321 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:08.265527 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb83c1b-8eab-465c-a628-4ed4f96335db None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:08.298119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05959f37-0274-466a-9506-121667a71add None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:08.298705 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05959f37-0274-466a-9506-121667a71add None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:08.299008 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05959f37-0274-466a-9506-121667a71add None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:08.299279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05959f37-0274-466a-9506-121667a71add None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:08.299980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05959f37-0274-466a-9506-121667a71add None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:08.300283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05959f37-0274-466a-9506-121667a71add None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:08.301670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05959f37-0274-466a-9506-121667a71add None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:08.318661 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 633/1254] 10.222.0.15 () {56 vars in 1143 bytes} [Tue Sep 13 01:09:08 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:09:08.324318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811dfd3b-2a55-412a-8e73-0440ef95125a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:08.422670 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811dfd3b-2a55-412a-8e73-0440ef95125a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:08.455946 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3410400-722f-4e09-a531-161d61c6c050 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:08.456547 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3410400-722f-4e09-a531-161d61c6c050 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:08.456825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3410400-722f-4e09-a531-161d61c6c050 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:08.457093 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3410400-722f-4e09-a531-161d61c6c050 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:08.509686 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:08.522419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3410400-722f-4e09-a531-161d61c6c050 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:08.523868 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3410400-722f-4e09-a531-161d61c6c050 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:08.617933 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 622/1255] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:09:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 01:09:08.624316 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05959f37-0274-466a-9506-121667a71add None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:08.725191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05959f37-0274-466a-9506-121667a71add None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:08.738147 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:08.758567 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40f1806-685a-416f-aec5-149a49fc9f81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:08.759209 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40f1806-685a-416f-aec5-149a49fc9f81 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:08.759486 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40f1806-685a-416f-aec5-149a49fc9f81 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:08.759806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40f1806-685a-416f-aec5-149a49fc9f81 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:08.769795 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40f1806-685a-416f-aec5-149a49fc9f81 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:08.771237 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40f1806-685a-416f-aec5-149a49fc9f81 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:08.785669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-c40f1806-685a-416f-aec5-149a49fc9f81 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:09:08.785669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:08.785669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:08.785669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:08.785669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:08.785669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:08.785669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:08.785669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:08.785669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:08.785669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:08.785669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:08.785669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:08.785669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:09:08.785669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:09:08.785669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:09:08.785669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:09:08.786767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:08.786767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:08.786767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:08.786767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:08.786767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:08.786767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:08.786767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:08.786767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:08.786767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:08.786767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:08.786767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:08.786767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:08.786767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:08.786767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:08.786767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:08.786767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:08.786767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:09:08.787936 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:09:08.787936 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:09:08.787936 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:09:08.787936 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 634/1256] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:09:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:08.792357 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3410400-722f-4e09-a531-161d61c6c050 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:08.875531 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:08.887816 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3410400-722f-4e09-a531-161d61c6c050 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:08.920418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f57537-b01d-4f2b-aab7-ee5ea02328f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:08.921012 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f57537-b01d-4f2b-aab7-ee5ea02328f9 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:08.921289 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f57537-b01d-4f2b-aab7-ee5ea02328f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:08.921559 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f57537-b01d-4f2b-aab7-ee5ea02328f9 None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:08.922185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f57537-b01d-4f2b-aab7-ee5ea02328f9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:08.922482 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f57537-b01d-4f2b-aab7-ee5ea02328f9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:08.923852 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f57537-b01d-4f2b-aab7-ee5ea02328f9 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:08.934548 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 623/1257] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:09:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:08.940234 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40f1806-685a-416f-aec5-149a49fc9f81 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:09.036390 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40f1806-685a-416f-aec5-149a49fc9f81 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:09.071213 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2a04d9-7474-4983-82cd-1f97b093dd19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:09.071892 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2a04d9-7474-4983-82cd-1f97b093dd19 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:09.072242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2a04d9-7474-4983-82cd-1f97b093dd19 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:09.072600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2a04d9-7474-4983-82cd-1f97b093dd19 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:09.081217 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2a04d9-7474-4983-82cd-1f97b093dd19 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:09.082993 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2a04d9-7474-4983-82cd-1f97b093dd19 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:09.096825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-fa2a04d9-7474-4983-82cd-1f97b093dd19 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:09:09.096825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:09.096825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:09.096825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:09.096825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:09.096825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:09.096825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:09.096825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:09.096825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:09.096825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:09.096825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:09.096825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:09.096825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:09:09.096825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:09:09.096825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:09:09.097854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:09:09.097854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:09.097854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:09.097854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:09.097854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:09.097854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:09.097854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:09.097854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:09.097854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:09.097854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:09.097854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:09.097854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:09.097854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:09.097854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:09.097854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:09.097854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:09.097854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:09.098738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:09:09.098738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:09:09.098738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:09:09.098738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:09:09.098738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:09:09.098738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:09:09.098738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:09:09.098738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:09:09.098738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 635/1258] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:09:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:09.105231 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f57537-b01d-4f2b-aab7-ee5ea02328f9 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:09.205402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f57537-b01d-4f2b-aab7-ee5ea02328f9 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:09.237691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947dd148-2a0e-4e18-8e14-b53e227822e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:09.238288 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947dd148-2a0e-4e18-8e14-b53e227822e0 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:09.238569 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947dd148-2a0e-4e18-8e14-b53e227822e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:09.238871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947dd148-2a0e-4e18-8e14-b53e227822e0 None admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:09.239559 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947dd148-2a0e-4e18-8e14-b53e227822e0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:09.239865 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947dd148-2a0e-4e18-8e14-b53e227822e0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:09.241278 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947dd148-2a0e-4e18-8e14-b53e227822e0 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:09.253385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 624/1259] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:09:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:09.259337 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2a04d9-7474-4983-82cd-1f97b093dd19 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:09.284875 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:09.357982 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2a04d9-7474-4983-82cd-1f97b093dd19 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:09.390585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9558e83-3776-47c7-be58-6af8ed2f49c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:09.391200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9558e83-3776-47c7-be58-6af8ed2f49c1 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:09.391478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9558e83-3776-47c7-be58-6af8ed2f49c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:09.391749 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9558e83-3776-47c7-be58-6af8ed2f49c1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:09.392457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9558e83-3776-47c7-be58-6af8ed2f49c1 None admin] RBAC: Adding query filter params (role.id=66a36c858aab40688080eed6ceace3a0, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=8eaff1b843a145a49994dcf089f44315) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:09.392766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9558e83-3776-47c7-be58-6af8ed2f49c1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:09.394171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9558e83-3776-47c7-be58-6af8ed2f49c1 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:09.399377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 636/1260] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:09:09 2022] GET /identity/v3/role_assignments?user.id=8eaff1b843a145a49994dcf089f44315&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=66a36c858aab40688080eed6ceace3a0 => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:11.507017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32be80b0-af4d-4068-b638-f2217f7c28d7 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:11.507558 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32be80b0-af4d-4068-b638-f2217f7c28d7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:11.507558 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32be80b0-af4d-4068-b638-f2217f7c28d7 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:11.508882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 625/1261] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:09:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:09:11.513929 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf77246-5928-462f-a6f3-cb4d5880bf24 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:11.514304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf77246-5928-462f-a6f3-cb4d5880bf24 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:11.514477 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf77246-5928-462f-a6f3-cb4d5880bf24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:11.532810 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-1bf77246-5928-462f-a6f3-cb4d5880bf24 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:09:11.533967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1bf77246-5928-462f-a6f3-cb4d5880bf24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:09:11.598087 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 637/1262] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:09:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:09:11.866970 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32be80b0-af4d-4068-b638-f2217f7c28d7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:11.878319 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:11.968909 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32be80b0-af4d-4068-b638-f2217f7c28d7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:12.001090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ba14b8-6f48-4343-87f6-eda5475bdd60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:12.001712 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ba14b8-6f48-4343-87f6-eda5475bdd60 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:12.001989 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ba14b8-6f48-4343-87f6-eda5475bdd60 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:12.002259 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ba14b8-6f48-4343-87f6-eda5475bdd60 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:12.012466 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ba14b8-6f48-4343-87f6-eda5475bdd60 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:09:12.017221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-f3ba14b8-6f48-4343-87f6-eda5475bdd60 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:09:12.017221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:12.017221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:12.017221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:12.017221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:12.017221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:12.017221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:12.017221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:12.017221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:12.017221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:12.017221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:12.017221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:12.017221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:09:12.017221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:09:12.017221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:09:12.017221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:09:12.018318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:09:12.018318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:09:12.018318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:09:12.018318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:09:12.018318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:12.018318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:12.018318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:12.018318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:12.018318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:12.018318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:12.018318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:12.018318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:12.018318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:12.018318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:12.018318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:12.018318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:12.019428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:12.019428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:12.019428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:12.019428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:12.019428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:09:12.019428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:09:12.019428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:09:12.019428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:09:12.019428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:09:12.019428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:09:12.019428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:09:12.019428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:09:12.019428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 626/1263] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:09:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:12.024369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf77246-5928-462f-a6f3-cb4d5880bf24 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:12.042663 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:12.126212 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf77246-5928-462f-a6f3-cb4d5880bf24 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:12.159427 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e82afd2-cfa9-4359-9454-29eedd94d5ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:12.160024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e82afd2-cfa9-4359-9454-29eedd94d5ec None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:12.160298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e82afd2-cfa9-4359-9454-29eedd94d5ec None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:12.160563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e82afd2-cfa9-4359-9454-29eedd94d5ec None admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:12.161208 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e82afd2-cfa9-4359-9454-29eedd94d5ec None admin] RBAC: Adding query filter params (name=admin) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:12.161506 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e82afd2-cfa9-4359-9454-29eedd94d5ec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:12.162840 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e82afd2-cfa9-4359-9454-29eedd94d5ec None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:12.172531 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 638/1264] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:09:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:12.178350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ba14b8-6f48-4343-87f6-eda5475bdd60 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:12.245462 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:12.278738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ba14b8-6f48-4343-87f6-eda5475bdd60 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:12.312291 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f552817-0f4a-43f0-932f-8c5c02af6d6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:12.312905 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f552817-0f4a-43f0-932f-8c5c02af6d6b None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:12.313180 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f552817-0f4a-43f0-932f-8c5c02af6d6b None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:12.313446 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f552817-0f4a-43f0-932f-8c5c02af6d6b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:12.323040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f552817-0f4a-43f0-932f-8c5c02af6d6b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:12.325051 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f552817-0f4a-43f0-932f-8c5c02af6d6b None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:12.339015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-4f552817-0f4a-43f0-932f-8c5c02af6d6b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:09:12.339015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:12.339015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:12.339015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:12.339015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:12.339015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:12.339015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:12.339015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:12.339015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:12.339015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:12.339015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:12.339015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:12.339015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:09:12.339015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:09:12.339015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:09:12.339015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:09:12.340153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:12.340153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:12.340153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:12.340153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:12.340153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:12.340153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:12.340153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:12.340153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:12.340153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:12.340153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:12.340153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:12.340153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:12.340153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:12.340153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:12.340153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:12.340153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:12.340153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:09:12.341108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:09:12.341108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:09:12.341108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:09:12.341108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 627/1265] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:09:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:12.345909 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e82afd2-cfa9-4359-9454-29eedd94d5ec None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:12.426174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:12.445871 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e82afd2-cfa9-4359-9454-29eedd94d5ec None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:12.478011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f8b20e-c02a-4f69-aff7-a67837c7537d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:12.478606 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f8b20e-c02a-4f69-aff7-a67837c7537d None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:12.478915 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f8b20e-c02a-4f69-aff7-a67837c7537d None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:12.479185 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f8b20e-c02a-4f69-aff7-a67837c7537d None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:12.479812 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f8b20e-c02a-4f69-aff7-a67837c7537d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:12.480112 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f8b20e-c02a-4f69-aff7-a67837c7537d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:12.481992 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f8b20e-c02a-4f69-aff7-a67837c7537d None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:12.493210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 639/1266] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:09:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:12.499618 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f552817-0f4a-43f0-932f-8c5c02af6d6b None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:12.596385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f552817-0f4a-43f0-932f-8c5c02af6d6b None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:12.615508 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:12.629833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b06700-d335-44e4-95ca-9e07b94d4d66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:12.630460 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b06700-d335-44e4-95ca-9e07b94d4d66 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:12.630740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b06700-d335-44e4-95ca-9e07b94d4d66 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:12.631041 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b06700-d335-44e4-95ca-9e07b94d4d66 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:12.638476 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b06700-d335-44e4-95ca-9e07b94d4d66 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:12.640007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b06700-d335-44e4-95ca-9e07b94d4d66 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:12.651305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-84b06700-d335-44e4-95ca-9e07b94d4d66 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 13 01:09:12.651305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:12.651305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:12.651305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:12.651305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:12.651305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:12.651305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:12.651305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:12.651305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:12.651305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:12.651305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:12.651305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:12.651305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:09:12.651305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:09:12.651305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:09:12.651305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:09:12.652515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:12.652515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:12.652515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:09:12.652515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:09:12.652515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:09:12.652515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:09:12.652515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:12.652515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:12.652515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:12.652515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:12.652515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:12.652515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:12.652515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:12.652515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:12.652515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:12.652515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:12.652515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:12.652515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:12.654098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:12.654098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:12.654098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:09:12.654098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:09:12.654098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:09:12.654098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:09:12.654098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:09:12.654098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:09:12.654098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 13 01:09:12.654098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:09:12.654098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 628/1267] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:09:12 2022] GET /identity/v3/users/placement => generated 87 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:12.658234 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f8b20e-c02a-4f69-aff7-a67837c7537d None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:12.755457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f8b20e-c02a-4f69-aff7-a67837c7537d None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:12.788066 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4e529f-4d83-4c4c-bd4c-cfd11cbbfcb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:12.788658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4e529f-4d83-4c4c-bd4c-cfd11cbbfcb3 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:12.788934 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4e529f-4d83-4c4c-bd4c-cfd11cbbfcb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:12.789203 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4e529f-4d83-4c4c-bd4c-cfd11cbbfcb3 None admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:12.789905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4e529f-4d83-4c4c-bd4c-cfd11cbbfcb3 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:12.790207 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4e529f-4d83-4c4c-bd4c-cfd11cbbfcb3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:12.791642 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4e529f-4d83-4c4c-bd4c-cfd11cbbfcb3 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:12.800670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:12.810290 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 640/1268] 10.222.0.15 () {56 vars in 1143 bytes} [Tue Sep 13 01:09:12 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:12.816017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b06700-d335-44e4-95ca-9e07b94d4d66 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:12.914248 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b06700-d335-44e4-95ca-9e07b94d4d66 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:12.947946 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499d74a2-eb03-46ba-b8b4-d008ee103106 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:12.948561 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499d74a2-eb03-46ba-b8b4-d008ee103106 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:12.948835 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499d74a2-eb03-46ba-b8b4-d008ee103106 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:12.949103 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499d74a2-eb03-46ba-b8b4-d008ee103106 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:12.959734 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499d74a2-eb03-46ba-b8b4-d008ee103106 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:12.961156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499d74a2-eb03-46ba-b8b4-d008ee103106 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:12.975622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-499d74a2-eb03-46ba-b8b4-d008ee103106 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:09:12.975622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:12.975622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:12.975622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:12.975622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:12.975622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:12.975622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:12.975622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:12.975622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:12.975622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:12.975622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:12.975622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:12.975622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:09:12.975622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:09:12.975622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:09:12.975622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:09:12.976539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:12.976539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:12.976539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:12.976539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:12.976539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:12.976539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:12.976539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:12.976539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:12.976539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:12.976539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:12.976539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:12.976539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:12.976539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:12.976539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:12.976539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:12.976539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:12.976539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:09:12.977482 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:09:12.977482 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:09:12.977482 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:09:12.977482 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 629/1269] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:09:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:12.983096 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4e529f-4d83-4c4c-bd4c-cfd11cbbfcb3 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:13.085326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4e529f-4d83-4c4c-bd4c-cfd11cbbfcb3 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:13.120113 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62610960-1e92-4873-8e6a-614de03a0f06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:13.120710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62610960-1e92-4873-8e6a-614de03a0f06 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:13.120987 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62610960-1e92-4873-8e6a-614de03a0f06 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:13.121259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62610960-1e92-4873-8e6a-614de03a0f06 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:13.121885 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62610960-1e92-4873-8e6a-614de03a0f06 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:13.122184 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62610960-1e92-4873-8e6a-614de03a0f06 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:13.123563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62610960-1e92-4873-8e6a-614de03a0f06 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:13.134455 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 641/1270] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:09:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:13.140147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499d74a2-eb03-46ba-b8b4-d008ee103106 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:13.172663 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:13.241419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499d74a2-eb03-46ba-b8b4-d008ee103106 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:13.280819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6850158-bd9b-44e7-aeb0-ac920d953082 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:13.281456 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6850158-bd9b-44e7-aeb0-ac920d953082 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:13.281731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6850158-bd9b-44e7-aeb0-ac920d953082 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:13.281998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6850158-bd9b-44e7-aeb0-ac920d953082 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:13.289670 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6850158-bd9b-44e7-aeb0-ac920d953082 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:13.291123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6850158-bd9b-44e7-aeb0-ac920d953082 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:13.302056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-c6850158-bd9b-44e7-aeb0-ac920d953082 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:09:13.302056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:13.302056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:13.302056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:13.302056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:13.302056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:13.302056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:13.302056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:13.302056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:13.302056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:13.302056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:13.302056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:13.302056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:09:13.302056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:09:13.302056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:09:13.303145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:09:13.303145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:13.303145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:13.303145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:13.303145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:13.303145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:13.303145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:13.303145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:13.303145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:13.303145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:13.303145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:13.303145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:13.303145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:13.303145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:13.303145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:13.303145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:13.303145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:13.304066 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:09:13.304066 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:09:13.304066 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:09:13.304066 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:09:13.304066 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:09:13.304066 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:09:13.304066 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:09:13.304066 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:09:13.304066 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 630/1271] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:09:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:13.308998 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62610960-1e92-4873-8e6a-614de03a0f06 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:13.367589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:13.409861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62610960-1e92-4873-8e6a-614de03a0f06 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:13.442663 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b9d9de-adde-4f13-9eeb-8610454a739a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:13.443292 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b9d9de-adde-4f13-9eeb-8610454a739a None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:13.443569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b9d9de-adde-4f13-9eeb-8610454a739a None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:13.443834 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b9d9de-adde-4f13-9eeb-8610454a739a None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:13.444509 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b9d9de-adde-4f13-9eeb-8610454a739a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:13.444810 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b9d9de-adde-4f13-9eeb-8610454a739a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:13.446219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b9d9de-adde-4f13-9eeb-8610454a739a None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:13.458407 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 642/1272] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:09:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:13.464435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6850158-bd9b-44e7-aeb0-ac920d953082 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:13.553647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:13.563770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6850158-bd9b-44e7-aeb0-ac920d953082 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:13.604368 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f240f3b-98ad-4c78-914c-6c3389eb5f09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:13.605066 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f240f3b-98ad-4c78-914c-6c3389eb5f09 None admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:13.605314 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f240f3b-98ad-4c78-914c-6c3389eb5f09 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:13.605582 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f240f3b-98ad-4c78-914c-6c3389eb5f09 None admin] PATH_INFO: `/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/8eaff1b843a145a49994dcf089f44315/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:13.607524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f240f3b-98ad-4c78-914c-6c3389eb5f09 None admin] RBAC: Authorizing `identity:create_grant(project_id=e5c1519fdae742f4a25eb7cd5476a8cb, user_id=8eaff1b843a145a49994dcf089f44315, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:13.608952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f240f3b-98ad-4c78-914c-6c3389eb5f09 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:13.622548 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 631/1273] 10.222.0.15 () {58 vars in 1439 bytes} [Tue Sep 13 01:09:13 2022] PUT /identity/v3/projects/e5c1519fdae742f4a25eb7cd5476a8cb/users/8eaff1b843a145a49994dcf089f44315/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:09:15.734961 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1a8b1d-8926-4889-a405-c971474e1e23 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:15.735478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1a8b1d-8926-4889-a405-c971474e1e23 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:15.735478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1a8b1d-8926-4889-a405-c971474e1e23 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:15.736832 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 643/1274] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:09:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:09:15.742050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e799b3c-f09d-4276-a9e3-8427f76be387 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:15.742427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e799b3c-f09d-4276-a9e3-8427f76be387 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:15.742591 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e799b3c-f09d-4276-a9e3-8427f76be387 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:15.760873 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7e799b3c-f09d-4276-a9e3-8427f76be387 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:09:15.762098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7e799b3c-f09d-4276-a9e3-8427f76be387 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:09:15.826078 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 632/1275] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:09:15 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:09:16.099368 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1a8b1d-8926-4889-a405-c971474e1e23 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:16.207150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1a8b1d-8926-4889-a405-c971474e1e23 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:16.224840 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:16.241345 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92941c4d-4808-4d9a-bffe-a7c1a6603fed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:16.241958 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92941c4d-4808-4d9a-bffe-a7c1a6603fed None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:16.242229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92941c4d-4808-4d9a-bffe-a7c1a6603fed None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:16.242493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92941c4d-4808-4d9a-bffe-a7c1a6603fed None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:16.252712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92941c4d-4808-4d9a-bffe-a7c1a6603fed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 01:09:16.257301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-92941c4d-4808-4d9a-bffe-a7c1a6603fed None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:09:16.257301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:16.257301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:16.257301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:16.257301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:16.257301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:16.257301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:16.257301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:16.257301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:16.257301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:16.257301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:16.257301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:16.257301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 01:09:16.257301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 01:09:16.257301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 01:09:16.257301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 01:09:16.258369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 01:09:16.258369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 01:09:16.258369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 01:09:16.258369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 01:09:16.258369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:16.258369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:16.258369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:16.258369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:16.258369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:16.258369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:16.258369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:16.258369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:16.258369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:16.258369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:16.258369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:16.258369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:16.259481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:16.259481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:16.259481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:16.259481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:16.259481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 01:09:16.259481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 01:09:16.259481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 01:09:16.259481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 01:09:16.259481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 01:09:16.259481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 01:09:16.259481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 01:09:16.259481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:09:16.259481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 644/1276] 10.222.0.15 () {56 vars in 1096 bytes} [Tue Sep 13 01:09:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:16.264440 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e799b3c-f09d-4276-a9e3-8427f76be387 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:16.341818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:16.360356 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e799b3c-f09d-4276-a9e3-8427f76be387 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:16.392840 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e834a6-bfbf-4c21-a6f1-b42ee65b9443 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:16.393434 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e834a6-bfbf-4c21-a6f1-b42ee65b9443 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:16.393712 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e834a6-bfbf-4c21-a6f1-b42ee65b9443 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:16.393986 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e834a6-bfbf-4c21-a6f1-b42ee65b9443 None admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:16.394638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e834a6-bfbf-4c21-a6f1-b42ee65b9443 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:16.394966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e834a6-bfbf-4c21-a6f1-b42ee65b9443 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:16.396319 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e834a6-bfbf-4c21-a6f1-b42ee65b9443 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:16.404942 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 633/1277] 10.222.0.15 () {56 vars in 1099 bytes} [Tue Sep 13 01:09:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:16.410744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92941c4d-4808-4d9a-bffe-a7c1a6603fed None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:16.507543 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92941c4d-4808-4d9a-bffe-a7c1a6603fed None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:16.539772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5bbe7c-a1e9-48b4-8212-8f7e85449678 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:16.540381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5bbe7c-a1e9-48b4-8212-8f7e85449678 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:16.540656 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5bbe7c-a1e9-48b4-8212-8f7e85449678 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:16.540926 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5bbe7c-a1e9-48b4-8212-8f7e85449678 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:16.551589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5bbe7c-a1e9-48b4-8212-8f7e85449678 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:16.553004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5bbe7c-a1e9-48b4-8212-8f7e85449678 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:16.567446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-1c5bbe7c-a1e9-48b4-8212-8f7e85449678 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:09:16.567446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:16.567446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:16.567446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:16.567446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:16.567446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:16.567446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:16.567446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:16.567446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:16.567446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:16.567446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:16.567446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:16.567446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:09:16.567446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:09:16.567446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:09:16.567446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:09:16.568612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:16.568612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:16.568612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:16.568612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:16.568612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:16.568612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:16.568612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:16.568612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:16.568612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:16.568612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:16.568612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:16.568612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:16.568612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:16.568612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:16.568612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:16.568612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:16.568612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:09:16.569792 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:09:16.569792 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:09:16.569792 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:09:16.569792 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 645/1278] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:09:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:16.574179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e834a6-bfbf-4c21-a6f1-b42ee65b9443 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:16.671877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e834a6-bfbf-4c21-a6f1-b42ee65b9443 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:16.703558 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678edca3-f7c5-4911-b25a-00a2e3ef70cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:16.704144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678edca3-f7c5-4911-b25a-00a2e3ef70cb None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:16.704421 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678edca3-f7c5-4911-b25a-00a2e3ef70cb None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:16.704689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678edca3-f7c5-4911-b25a-00a2e3ef70cb None admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:16.705318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678edca3-f7c5-4911-b25a-00a2e3ef70cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:16.705614 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678edca3-f7c5-4911-b25a-00a2e3ef70cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:16.706978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678edca3-f7c5-4911-b25a-00a2e3ef70cb None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:16.718977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 634/1279] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:09:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:16.724511 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5bbe7c-a1e9-48b4-8212-8f7e85449678 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:16.754595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:16.821974 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5bbe7c-a1e9-48b4-8212-8f7e85449678 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:16.854098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf6b9f7-41d5-4413-a5af-b57210f20e75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:16.854703 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf6b9f7-41d5-4413-a5af-b57210f20e75 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:16.855010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf6b9f7-41d5-4413-a5af-b57210f20e75 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:16.855283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf6b9f7-41d5-4413-a5af-b57210f20e75 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:16.862991 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf6b9f7-41d5-4413-a5af-b57210f20e75 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:16.864412 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf6b9f7-41d5-4413-a5af-b57210f20e75 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:16.875210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-6bf6b9f7-41d5-4413-a5af-b57210f20e75 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 13 01:09:16.875210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:16.875210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:16.875210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:16.875210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:16.875210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:16.875210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:16.875210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:16.875210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:16.875210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:16.875210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:16.875210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:16.875210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 01:09:16.875210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 01:09:16.875210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 01:09:16.875210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 01:09:16.876302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:16.876302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:16.876302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 01:09:16.876302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:09:16.876302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 01:09:16.876302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 01:09:16.876302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:16.876302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:16.876302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:16.876302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:16.876302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:16.876302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:16.876302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:16.876302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:16.876302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:16.876302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:16.876302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:16.876302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:16.877493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:16.877493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:16.877493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 01:09:16.877493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 01:09:16.877493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 01:09:16.877493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 01:09:16.877493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 01:09:16.877493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 01:09:16.877493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 13 01:09:16.877493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:09:16.877493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 646/1280] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:09:16 2022] GET /identity/v3/users/placement => generated 87 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:16.882148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678edca3-f7c5-4911-b25a-00a2e3ef70cb None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:16.885871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:16.982833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678edca3-f7c5-4911-b25a-00a2e3ef70cb None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:17.015748 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b16a74-0e78-4030-9163-3b7e8b50f607 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:17.016346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b16a74-0e78-4030-9163-3b7e8b50f607 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:17.016624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b16a74-0e78-4030-9163-3b7e8b50f607 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:17.016895 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b16a74-0e78-4030-9163-3b7e8b50f607 None admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:17.017606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31b16a74-0e78-4030-9163-3b7e8b50f607 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:17.017907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31b16a74-0e78-4030-9163-3b7e8b50f607 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:17.019353 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31b16a74-0e78-4030-9163-3b7e8b50f607 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:17.036402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 635/1281] 10.222.0.15 () {56 vars in 1143 bytes} [Tue Sep 13 01:09:16 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:17.042060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf6b9f7-41d5-4413-a5af-b57210f20e75 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:17.130929 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:17.141396 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf6b9f7-41d5-4413-a5af-b57210f20e75 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:17.173408 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef29653-5b8b-4b48-9788-007a2e89d30f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:17.173993 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef29653-5b8b-4b48-9788-007a2e89d30f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:17.174267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef29653-5b8b-4b48-9788-007a2e89d30f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:17.174532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef29653-5b8b-4b48-9788-007a2e89d30f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:17.238429 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef29653-5b8b-4b48-9788-007a2e89d30f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:17.239813 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef29653-5b8b-4b48-9788-007a2e89d30f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:17.323755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:17.334283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 647/1282] 10.222.0.15 () {58 vars in 1282 bytes} [Tue Sep 13 01:09:17 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 01:09:17.339994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b16a74-0e78-4030-9163-3b7e8b50f607 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:17.398689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:17.436958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b16a74-0e78-4030-9163-3b7e8b50f607 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:17.470146 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc03787f-21a0-413f-a0f4-1d47b9df521a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:17.470760 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc03787f-21a0-413f-a0f4-1d47b9df521a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:17.471065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc03787f-21a0-413f-a0f4-1d47b9df521a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:17.471330 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc03787f-21a0-413f-a0f4-1d47b9df521a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:17.480854 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc03787f-21a0-413f-a0f4-1d47b9df521a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:17.482270 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc03787f-21a0-413f-a0f4-1d47b9df521a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:17.496747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-fc03787f-21a0-413f-a0f4-1d47b9df521a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:09:17.496747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:17.496747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:17.496747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:17.496747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:17.496747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:17.496747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:17.496747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:17.496747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:17.496747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:17.496747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:17.496747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:17.496747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 01:09:17.496747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 01:09:17.496747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 01:09:17.496747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 01:09:17.497861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:17.497861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:17.497861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:17.497861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:17.497861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:17.497861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:17.497861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:17.497861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:17.497861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:17.497861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:17.497861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:17.497861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:17.497861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:17.497861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:17.497861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:17.497861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:17.497861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 01:09:17.499092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 01:09:17.499092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 01:09:17.499092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:09:17.499092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 636/1283] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:09:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:17.503575 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef29653-5b8b-4b48-9788-007a2e89d30f None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:17.598662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef29653-5b8b-4b48-9788-007a2e89d30f None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:17.631405 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773900d9-8439-46be-b8ea-2d9bf1d00374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:17.631995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773900d9-8439-46be-b8ea-2d9bf1d00374 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:17.632266 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773900d9-8439-46be-b8ea-2d9bf1d00374 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:17.632530 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773900d9-8439-46be-b8ea-2d9bf1d00374 None admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:17.633150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773900d9-8439-46be-b8ea-2d9bf1d00374 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:17.633446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773900d9-8439-46be-b8ea-2d9bf1d00374 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:17.634778 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773900d9-8439-46be-b8ea-2d9bf1d00374 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:17.646050 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 648/1284] 10.222.0.15 () {56 vars in 1109 bytes} [Tue Sep 13 01:09:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:17.651574 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc03787f-21a0-413f-a0f4-1d47b9df521a None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:17.747318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc03787f-21a0-413f-a0f4-1d47b9df521a None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:17.760088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:17.780875 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b1802a-4baf-469a-a2ae-25eea0126b06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:17.781471 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b1802a-4baf-469a-a2ae-25eea0126b06 None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:17.781744 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b1802a-4baf-469a-a2ae-25eea0126b06 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:17.782011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b1802a-4baf-469a-a2ae-25eea0126b06 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:17.788373 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b1802a-4baf-469a-a2ae-25eea0126b06 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:17.789783 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b1802a-4baf-469a-a2ae-25eea0126b06 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:17.800538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application [None req-63b1802a-4baf-469a-a2ae-25eea0126b06 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:09:17.800538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:17.800538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:17.800538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:17.800538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:17.800538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:17.800538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:17.800538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:17.800538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:17.800538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:17.800538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:17.800538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:17.800538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 01:09:17.800538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 01:09:17.800538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 01:09:17.801572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 01:09:17.801572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:17.801572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:17.801572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:17.801572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:17.801572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:17.801572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:17.801572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:17.801572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:17.801572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:17.801572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:17.801572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:17.801572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:17.801572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:17.801572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:17.801572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:17.801572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:17.802533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 01:09:17.802533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 01:09:17.802533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 01:09:17.802533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 01:09:17.802533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 01:09:17.802533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 01:09:17.802533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 01:09:17.802533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: ERROR keystone.server.flask.application Sep 13 01:09:17.802533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 637/1285] 10.222.0.15 () {56 vars in 1111 bytes} [Tue Sep 13 01:09:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:17.807248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773900d9-8439-46be-b8ea-2d9bf1d00374 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:17.855503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:17.903667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773900d9-8439-46be-b8ea-2d9bf1d00374 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:17.935084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86eba9d7-7d0b-44ee-b8e5-02358eb57285 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:17.935671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eba9d7-7d0b-44ee-b8e5-02358eb57285 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:17.935942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eba9d7-7d0b-44ee-b8e5-02358eb57285 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:17.936204 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eba9d7-7d0b-44ee-b8e5-02358eb57285 None admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:17.936867 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86eba9d7-7d0b-44ee-b8e5-02358eb57285 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:17.937163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86eba9d7-7d0b-44ee-b8e5-02358eb57285 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:17.938544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86eba9d7-7d0b-44ee-b8e5-02358eb57285 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:17.950379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 649/1286] 10.222.0.15 () {56 vars in 1148 bytes} [Tue Sep 13 01:09:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:17.956352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b1802a-4baf-469a-a2ae-25eea0126b06 None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:18.053669 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b1802a-4baf-469a-a2ae-25eea0126b06 None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:18.089722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3362af0-eead-46d4-83bd-40c37e72b11a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:18.090317 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3362af0-eead-46d4-83bd-40c37e72b11a None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:18.090595 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3362af0-eead-46d4-83bd-40c37e72b11a None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:18.090886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3362af0-eead-46d4-83bd-40c37e72b11a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:18.091599 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3362af0-eead-46d4-83bd-40c37e72b11a None admin] RBAC: Adding query filter params (role.id=66a36c858aab40688080eed6ceace3a0, scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb, user.id=8eaff1b843a145a49994dcf089f44315) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:18.091903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3362af0-eead-46d4-83bd-40c37e72b11a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:18.093313 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3362af0-eead-46d4-83bd-40c37e72b11a None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:18.098939 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 638/1287] 10.222.0.15 () {56 vars in 1374 bytes} [Tue Sep 13 01:09:17 2022] GET /identity/v3/role_assignments?user.id=8eaff1b843a145a49994dcf089f44315&scope.project.id=e5c1519fdae742f4a25eb7cd5476a8cb&role.id=66a36c858aab40688080eed6ceace3a0 => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:20.225686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5106ef-ccdd-45d6-b67a-4a832b17985c None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:20.226242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5106ef-ccdd-45d6-b67a-4a832b17985c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:20.226242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5106ef-ccdd-45d6-b67a-4a832b17985c None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:20.227587 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 650/1288] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:09:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:09:20.232818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d1e54f-8a53-4047-a842-ab5ea3295e50 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:20.233228 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d1e54f-8a53-4047-a842-ab5ea3295e50 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:20.233374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d1e54f-8a53-4047-a842-ab5ea3295e50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:20.251147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-89d1e54f-8a53-4047-a842-ab5ea3295e50 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:09:20.252335 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-89d1e54f-8a53-4047-a842-ab5ea3295e50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:09:20.282348 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:20.319478 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 639/1289] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:09:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:09:20.584270 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5106ef-ccdd-45d6-b67a-4a832b17985c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:20.688415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5106ef-ccdd-45d6-b67a-4a832b17985c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:20.698595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:20.721452 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38c6c94-aa35-4d81-a56e-c48bb86b0dcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:20.722071 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38c6c94-aa35-4d81-a56e-c48bb86b0dcb None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:20.722347 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38c6c94-aa35-4d81-a56e-c48bb86b0dcb None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:20.722616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38c6c94-aa35-4d81-a56e-c48bb86b0dcb None admin] PATH_INFO: `/v3/services/placement` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:20.723215 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38c6c94-aa35-4d81-a56e-c48bb86b0dcb None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:20.724559 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38c6c94-aa35-4d81-a56e-c48bb86b0dcb None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:20.733498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-f38c6c94-aa35-4d81-a56e-c48bb86b0dcb None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 01:09:20.733498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:20.733498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:09:20.733498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:09:20.733498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:09:20.733498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:09:20.733498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:09:20.733498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:09:20.733498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 01:09:20.733498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:09:20.733498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:09:20.733498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:09:20.733498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:20.733498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:20.733498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:20.733498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:20.733498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:20.734678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:20.734678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:20.734678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:20.734678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:20.734678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:20.734678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:20.734678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:09:20.734678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:09:20.734678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:09:20.734678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:09:20.734678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:20.734678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:20.734678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:20.734678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:20.734678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:20.734678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:20.735744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:20.735744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:20.735744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:20.735744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:20.735744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:20.735744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:20.735744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:20.735744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:20.735744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:20.735744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:20.735744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:09:20.735744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:09:20.735744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 01:09:20.735744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:09:20.735744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 651/1290] 10.222.0.15 () {56 vars in 1117 bytes} [Tue Sep 13 01:09:20 2022] GET /identity/v3/services/placement => generated 90 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:20.740544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d1e54f-8a53-4047-a842-ab5ea3295e50 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:20.835130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d1e54f-8a53-4047-a842-ab5ea3295e50 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:20.867278 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818e0e33-f4bd-4b24-8418-64d907b0062d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:20.867864 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818e0e33-f4bd-4b24-8418-64d907b0062d None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:20.868139 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818e0e33-f4bd-4b24-8418-64d907b0062d None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:20.868406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818e0e33-f4bd-4b24-8418-64d907b0062d None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:20.869015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818e0e33-f4bd-4b24-8418-64d907b0062d None admin] RBAC: Adding query filter params (name=placement) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:20.869313 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818e0e33-f4bd-4b24-8418-64d907b0062d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:20.870649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818e0e33-f4bd-4b24-8418-64d907b0062d None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:20.875810 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 640/1291] 10.222.0.15 () {56 vars in 1116 bytes} [Tue Sep 13 01:09:20 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:20.881299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38c6c94-aa35-4d81-a56e-c48bb86b0dcb None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:20.975886 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38c6c94-aa35-4d81-a56e-c48bb86b0dcb None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:21.007941 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0b4989-b84f-4e60-bc46-06826d40a38f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:21.008513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0b4989-b84f-4e60-bc46-06826d40a38f None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:21.008788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0b4989-b84f-4e60-bc46-06826d40a38f None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:21.009055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0b4989-b84f-4e60-bc46-06826d40a38f None admin] PATH_INFO: `/v3/services` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:21.009656 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0b4989-b84f-4e60-bc46-06826d40a38f None admin] RBAC: Adding query filter params (type=placement) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:21.009951 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0b4989-b84f-4e60-bc46-06826d40a38f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:21.011322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0b4989-b84f-4e60-bc46-06826d40a38f None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:21.015513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 652/1292] 10.222.0.15 () {56 vars in 1116 bytes} [Tue Sep 13 01:09:20 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:23.115831 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e88a0c-d0b2-494e-ac11-bf21dac98310 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:23.116366 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e88a0c-d0b2-494e-ac11-bf21dac98310 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:23.116366 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e88a0c-d0b2-494e-ac11-bf21dac98310 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:23.117700 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 641/1293] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:09:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:09:23.123452 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc672ef4-555d-40a9-b465-cc394085807f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:23.123871 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc672ef4-555d-40a9-b465-cc394085807f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:23.123975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc672ef4-555d-40a9-b465-cc394085807f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:23.141616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-fc672ef4-555d-40a9-b465-cc394085807f None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:09:23.142771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-fc672ef4-555d-40a9-b465-cc394085807f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:09:23.171231 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:23.207580 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 653/1294] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:09:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:09:23.470606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e88a0c-d0b2-494e-ac11-bf21dac98310 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:23.501310 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:23.569525 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e88a0c-d0b2-494e-ac11-bf21dac98310 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:23.602710 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d01908a-4eb0-47a8-8d8c-e8cf1139f308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:23.603329 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d01908a-4eb0-47a8-8d8c-e8cf1139f308 None admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:23.603610 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d01908a-4eb0-47a8-8d8c-e8cf1139f308 None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:23.603883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d01908a-4eb0-47a8-8d8c-e8cf1139f308 None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:23.604527 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d01908a-4eb0-47a8-8d8c-e8cf1139f308 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:23.606574 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d01908a-4eb0-47a8-8d8c-e8cf1139f308 None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:23.620438 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 642/1295] 10.222.0.15 () {60 vars in 1141 bytes} [Tue Sep 13 01:09:23 2022] POST /identity/v3/services => generated 246 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:09:25.736507 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd126607-fe51-462a-9469-ebba2464454f None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:25.737026 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd126607-fe51-462a-9469-ebba2464454f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:25.737026 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd126607-fe51-462a-9469-ebba2464454f None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:25.738367 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 654/1296] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:09:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:09:25.743396 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18b942e-f865-4f9a-87da-5b3130bf6631 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:25.743762 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18b942e-f865-4f9a-87da-5b3130bf6631 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:25.743944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18b942e-f865-4f9a-87da-5b3130bf6631 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:25.761494 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e18b942e-f865-4f9a-87da-5b3130bf6631 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:09:25.762659 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e18b942e-f865-4f9a-87da-5b3130bf6631 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:09:25.832333 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 643/1297] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:09:25 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:09:26.103675 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd126607-fe51-462a-9469-ebba2464454f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:26.202647 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd126607-fe51-462a-9469-ebba2464454f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:26.235564 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2aea29d-dcb9-4770-983b-67b32dfa89f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:26.236172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aea29d-dcb9-4770-983b-67b32dfa89f6 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:26.236443 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aea29d-dcb9-4770-983b-67b32dfa89f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:26.236708 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aea29d-dcb9-4770-983b-67b32dfa89f6 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:26.237270 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2aea29d-dcb9-4770-983b-67b32dfa89f6 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:26.238607 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2aea29d-dcb9-4770-983b-67b32dfa89f6 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:26.247437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-f2aea29d-dcb9-4770-983b-67b32dfa89f6 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 01:09:26.247437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:26.247437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:09:26.247437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:09:26.247437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:09:26.247437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:09:26.247437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:09:26.247437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:09:26.247437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 01:09:26.247437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:09:26.247437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:09:26.247437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:09:26.247437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:26.247437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:26.247437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:26.247437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:26.247437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:26.248549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:26.248549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:26.248549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:26.248549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:26.248549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:26.248549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:26.248549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:09:26.248549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:09:26.248549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:09:26.248549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:09:26.248549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:26.248549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:26.248549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:26.248549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:26.248549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:26.248549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:26.249825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:26.249825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:26.249825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:26.249825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:26.249825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:26.249825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:26.249825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:26.249825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:26.249825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:26.249825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:26.249825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:09:26.249825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:09:26.249825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 01:09:26.249825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:09:26.249825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 655/1298] 10.222.0.15 () {56 vars in 1117 bytes} [Tue Sep 13 01:09:26 2022] GET /identity/v3/services/placement => generated 90 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:26.254483 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18b942e-f865-4f9a-87da-5b3130bf6631 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:26.272719 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:26.351381 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18b942e-f865-4f9a-87da-5b3130bf6631 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:26.383832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c01776-0a41-4e81-8e2e-daf5366fc9de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:26.384370 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c01776-0a41-4e81-8e2e-daf5366fc9de None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:26.384648 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c01776-0a41-4e81-8e2e-daf5366fc9de None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:26.384919 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c01776-0a41-4e81-8e2e-daf5366fc9de None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:26.385582 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c01776-0a41-4e81-8e2e-daf5366fc9de None admin] RBAC: Adding query filter params (name=placement) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:26.385880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c01776-0a41-4e81-8e2e-daf5366fc9de None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:26.387866 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c01776-0a41-4e81-8e2e-daf5366fc9de None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:26.392828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 644/1299] 10.222.0.15 () {56 vars in 1116 bytes} [Tue Sep 13 01:09:26 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:26.398680 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2aea29d-dcb9-4770-983b-67b32dfa89f6 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:26.429936 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:26.495138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2aea29d-dcb9-4770-983b-67b32dfa89f6 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:26.527690 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c94b5e7-0f01-4846-8da3-77ada7f433b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:26.528277 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c94b5e7-0f01-4846-8da3-77ada7f433b6 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:26.528550 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c94b5e7-0f01-4846-8da3-77ada7f433b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:26.528815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c94b5e7-0f01-4846-8da3-77ada7f433b6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:26.529493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c94b5e7-0f01-4846-8da3-77ada7f433b6 None admin] RBAC: Adding query filter params (interface=public, service_id=9c7d8077d4d541688d77bc0527b22701, region_id=RegionOne) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:26.529792 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c94b5e7-0f01-4846-8da3-77ada7f433b6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:26.531158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c94b5e7-0f01-4846-8da3-77ada7f433b6 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:26.535929 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 656/1300] 10.222.0.15 () {56 vars in 1251 bytes} [Tue Sep 13 01:09:26 2022] GET /identity/v3/endpoints?service_id=9c7d8077d4d541688d77bc0527b22701&interface=public®ion_id=RegionOne => generated 194 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:26.541630 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c01776-0a41-4e81-8e2e-daf5366fc9de None admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:26.620179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:26.639670 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c01776-0a41-4e81-8e2e-daf5366fc9de None admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:26.676115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0b9b74-8a98-4857-88da-cf7ebd2f6fce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:26.676721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0b9b74-8a98-4857-88da-cf7ebd2f6fce None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:26.676999 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0b9b74-8a98-4857-88da-cf7ebd2f6fce None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:26.677266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0b9b74-8a98-4857-88da-cf7ebd2f6fce None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:26.677861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0b9b74-8a98-4857-88da-cf7ebd2f6fce None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:26.679346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0b9b74-8a98-4857-88da-cf7ebd2f6fce None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:26.686979 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 645/1301] 10.222.0.15 () {56 vars in 1087 bytes} [Tue Sep 13 01:09:26 2022] GET /identity/v3/services => generated 2191 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:09:28.798117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03646dd3-992f-4385-a030-248721107a0c None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:28.798643 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03646dd3-992f-4385-a030-248721107a0c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:28.798643 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03646dd3-992f-4385-a030-248721107a0c None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:28.800041 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 657/1302] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:09:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:09:28.805069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5b7a01-c4da-438f-929d-3396889ce671 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:28.805469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5b7a01-c4da-438f-929d-3396889ce671 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:28.805618 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5b7a01-c4da-438f-929d-3396889ce671 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:28.823647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7c5b7a01-c4da-438f-929d-3396889ce671 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:09:28.824826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7c5b7a01-c4da-438f-929d-3396889ce671 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:09:28.888830 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 646/1303] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:09:28 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 01:09:29.167536 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03646dd3-992f-4385-a030-248721107a0c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:29.261728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:29.268870 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03646dd3-992f-4385-a030-248721107a0c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:29.300941 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce880b5-46ab-42bd-8bf8-6e47eb78fcb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:29.301552 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce880b5-46ab-42bd-8bf8-6e47eb78fcb0 None admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:29.301823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce880b5-46ab-42bd-8bf8-6e47eb78fcb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:29.302088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce880b5-46ab-42bd-8bf8-6e47eb78fcb0 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:29.302651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce880b5-46ab-42bd-8bf8-6e47eb78fcb0 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:29.304028 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce880b5-46ab-42bd-8bf8-6e47eb78fcb0 None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:29.313211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application [None req-3ce880b5-46ab-42bd-8bf8-6e47eb78fcb0 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 01:09:29.313211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:29.313211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 01:09:29.313211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 01:09:29.313211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 01:09:29.313211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 01:09:29.313211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 01:09:29.313211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:09:29.313211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 01:09:29.313211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:09:29.313211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 01:09:29.313211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:09:29.313211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 01:09:29.313211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 01:09:29.313211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 01:09:29.313211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 01:09:29.313211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 01:09:29.314430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 01:09:29.314430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 01:09:29.314430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 01:09:29.314430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 01:09:29.314430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 01:09:29.314430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 01:09:29.314430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 01:09:29.314430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 01:09:29.314430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 01:09:29.314430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 01:09:29.314430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 01:09:29.314430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 01:09:29.314430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 01:09:29.314430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 01:09:29.314430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 01:09:29.314430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 01:09:29.315652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 01:09:29.315652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application with Lock( Sep 13 01:09:29.315652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 01:09:29.315652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self._enter() Sep 13 01:09:29.315652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 01:09:29.315652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 01:09:29.315652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 01:09:29.315652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application return self.creator() Sep 13 01:09:29.315652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 01:09:29.315652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application created_value = creator( Sep 13 01:09:29.315652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 01:09:29.315652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 01:09:29.315652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 01:09:29.315652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: ERROR keystone.server.flask.application Sep 13 01:09:29.315652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 658/1304] 10.222.0.15 () {56 vars in 1117 bytes} [Tue Sep 13 01:09:29 2022] GET /identity/v3/services/placement => generated 90 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 01:09:29.320264 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5b7a01-c4da-438f-929d-3396889ce671 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:29.393695 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:29.416488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5b7a01-c4da-438f-929d-3396889ce671 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:29.450263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0309152d-7dd7-43fa-b272-337d0049d41c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:29.450803 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0309152d-7dd7-43fa-b272-337d0049d41c None admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:29.451110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0309152d-7dd7-43fa-b272-337d0049d41c None admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:29.451382 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0309152d-7dd7-43fa-b272-337d0049d41c None admin] PATH_INFO: `/v3/services` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:29.451999 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0309152d-7dd7-43fa-b272-337d0049d41c None admin] RBAC: Adding query filter params (name=placement) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:09:29.452299 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0309152d-7dd7-43fa-b272-337d0049d41c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:29.453655 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0309152d-7dd7-43fa-b272-337d0049d41c None admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:29.458760 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 647/1305] 10.222.0.15 () {56 vars in 1116 bytes} [Tue Sep 13 01:09:29 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:09:29.464406 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce880b5-46ab-42bd-8bf8-6e47eb78fcb0 None admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:09:29.560079 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce880b5-46ab-42bd-8bf8-6e47eb78fcb0 None admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:09:29.592549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89054c42-eb09-4d14-94c0-b028a139f37b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:09:29.593133 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89054c42-eb09-4d14-94c0-b028a139f37b None admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:09:29.593409 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89054c42-eb09-4d14-94c0-b028a139f37b None admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:09:29.593685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89054c42-eb09-4d14-94c0-b028a139f37b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:09:29.594338 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89054c42-eb09-4d14-94c0-b028a139f37b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:09:29.595700 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89054c42-eb09-4d14-94c0-b028a139f37b None admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:09:29.614396 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 659/1306] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:09:29 2022] POST /identity/v3/endpoints => generated 331 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:09:39.456163 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=30 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:09:39.610529 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:10:33.308061 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf807de-2067-441c-81e2-d73cb338e75e None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:33.308631 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf807de-2067-441c-81e2-d73cb338e75e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:33.308631 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf807de-2067-441c-81e2-d73cb338e75e None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:33.310093 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 648/1307] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:10:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:10:33.316866 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283d23c4-03d8-4929-867d-ca6ea6daef6f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:33.317378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283d23c4-03d8-4929-867d-ca6ea6daef6f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:33.317378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283d23c4-03d8-4929-867d-ca6ea6daef6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:33.412038 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-283d23c4-03d8-4929-867d-ca6ea6daef6f None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:10:33.413821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-283d23c4-03d8-4929-867d-ca6ea6daef6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:33.538898 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 660/1308] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:10:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:10:33.821221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebabef55-b20a-478e-93b7-bee4b7441f53 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:33.821579 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebabef55-b20a-478e-93b7-bee4b7441f53 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:33.821937 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebabef55-b20a-478e-93b7-bee4b7441f53 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:33.823422 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 649/1309] 10.222.0.15 () {54 vars in 965 bytes} [Tue Sep 13 01:10:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:10:33.829401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dad5bc7-f5a7-4f3c-9d74-3c7e3ebe3ea7 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:33.829782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dad5bc7-f5a7-4f3c-9d74-3c7e3ebe3ea7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:33.830132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dad5bc7-f5a7-4f3c-9d74-3c7e3ebe3ea7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:33.915511 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9dad5bc7-f5a7-4f3c-9d74-3c7e3ebe3ea7 None None] MFA Rules not processed for user `09cd263f72484733adaaa0850ebfa609`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:10:33.917122 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9dad5bc7-f5a7-4f3c-9d74-3c7e3ebe3ea7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:33.939099 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:10:33.999631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 661/1310] 10.222.0.15 () {58 vars in 1062 bytes} [Tue Sep 13 01:10:33 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:10:34.009172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebabef55-b20a-478e-93b7-bee4b7441f53 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:10:34.010952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ebabef55-b20a-478e-93b7-bee4b7441f53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:34.132052 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebabef55-b20a-478e-93b7-bee4b7441f53 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:10:34.214212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf621fc0-1a8c-458a-b49e-9734ef945625 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:10:34.214832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf621fc0-1a8c-458a-b49e-9734ef945625 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:34.215137 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf621fc0-1a8c-458a-b49e-9734ef945625 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:34.215413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf621fc0-1a8c-458a-b49e-9734ef945625 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:34.217045 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-bf621fc0-1a8c-458a-b49e-9734ef945625 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:34.275692 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:10:34.341430 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf621fc0-1a8c-458a-b49e-9734ef945625 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:10:34.342915 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf621fc0-1a8c-458a-b49e-9734ef945625 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:10:34.365439 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 650/1311] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:10:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:10:36.818739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92f055c-7ff1-4a79-9295-c15e45325108 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:36.819998 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92f055c-7ff1-4a79-9295-c15e45325108 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:36.819998 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92f055c-7ff1-4a79-9295-c15e45325108 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:36.821204 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 662/1312] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:10:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:10:36.826534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c35763-bd93-4bf1-97bf-d435d19b8732 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:36.826964 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c35763-bd93-4bf1-97bf-d435d19b8732 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:36.827125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c35763-bd93-4bf1-97bf-d435d19b8732 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:36.898603 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f4c35763-bd93-4bf1-97bf-d435d19b8732 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:10:36.899860 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f4c35763-bd93-4bf1-97bf-d435d19b8732 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:36.996081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 651/1313] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:10:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:10:37.275336 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92f055c-7ff1-4a79-9295-c15e45325108 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:10:37.276925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a92f055c-7ff1-4a79-9295-c15e45325108 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:37.381123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92f055c-7ff1-4a79-9295-c15e45325108 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:10:37.456095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:10:37.458932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a63f8d4-4aab-47cb-8790-352a3271a0c0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:10:37.459613 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a63f8d4-4aab-47cb-8790-352a3271a0c0 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:37.460147 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a63f8d4-4aab-47cb-8790-352a3271a0c0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:37.460606 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a63f8d4-4aab-47cb-8790-352a3271a0c0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:37.462528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1a63f8d4-4aab-47cb-8790-352a3271a0c0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:37.573026 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a63f8d4-4aab-47cb-8790-352a3271a0c0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:10:37.574449 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a63f8d4-4aab-47cb-8790-352a3271a0c0 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:10:37.592622 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 663/1314] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:10:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:10:40.041219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e61072b-dd87-441a-9b05-7ae77450427b None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:40.041873 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e61072b-dd87-441a-9b05-7ae77450427b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:40.041873 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e61072b-dd87-441a-9b05-7ae77450427b None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:40.043136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 652/1315] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:10:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:10:40.048215 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d7a38d-6c79-407b-b588-5c0842fcdcec None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:40.048671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d7a38d-6c79-407b-b588-5c0842fcdcec None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:40.048774 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d7a38d-6c79-407b-b588-5c0842fcdcec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:40.114421 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-71d7a38d-6c79-407b-b588-5c0842fcdcec None None] MFA Rules not processed for user `1e80dcbdbef640078cfec6d97eabca95`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:10:40.115676 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-71d7a38d-6c79-407b-b588-5c0842fcdcec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:40.166315 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:10:40.211088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 664/1316] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:10:40 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:10:40.496388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e61072b-dd87-441a-9b05-7ae77450427b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:10:40.497926 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2e61072b-dd87-441a-9b05-7ae77450427b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:40.508295 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:10:40.600924 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e61072b-dd87-441a-9b05-7ae77450427b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:10:40.672955 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee87ba7a-8a0a-4657-bae9-faf92f484fc0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:10:40.673492 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee87ba7a-8a0a-4657-bae9-faf92f484fc0 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:40.673770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee87ba7a-8a0a-4657-bae9-faf92f484fc0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:40.674040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee87ba7a-8a0a-4657-bae9-faf92f484fc0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:40.675646 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ee87ba7a-8a0a-4657-bae9-faf92f484fc0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:40.735508 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:10:40.781434 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee87ba7a-8a0a-4657-bae9-faf92f484fc0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:10:40.782878 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee87ba7a-8a0a-4657-bae9-faf92f484fc0 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:10:40.800252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 653/1317] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:10:40 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:10:43.475079 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5c15cd-a7bb-4fc6-80f4-f685321a410e None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:43.475616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5c15cd-a7bb-4fc6-80f4-f685321a410e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:43.475789 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5c15cd-a7bb-4fc6-80f4-f685321a410e None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:43.477227 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 665/1318] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:10:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:10:43.482618 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a3c8f1-3c60-4348-9d23-e508cc428feb None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:43.483025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a3c8f1-3c60-4348-9d23-e508cc428feb None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:43.483200 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a3c8f1-3c60-4348-9d23-e508cc428feb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:43.548454 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d4a3c8f1-3c60-4348-9d23-e508cc428feb None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:10:43.549674 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d4a3c8f1-3c60-4348-9d23-e508cc428feb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:43.643524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 654/1319] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:10:43 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:10:43.927175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5c15cd-a7bb-4fc6-80f4-f685321a410e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:10:43.928823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0b5c15cd-a7bb-4fc6-80f4-f685321a410e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:44.038430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5c15cd-a7bb-4fc6-80f4-f685321a410e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:10:44.113388 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a077d8-43a2-498a-9830-f3f2b2bea8ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:10:44.113988 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a077d8-43a2-498a-9830-f3f2b2bea8ac service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:44.114261 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a077d8-43a2-498a-9830-f3f2b2bea8ac service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:44.114528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a077d8-43a2-498a-9830-f3f2b2bea8ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:44.116149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e6a077d8-43a2-498a-9830-f3f2b2bea8ac service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:44.122873 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:10:44.223466 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a077d8-43a2-498a-9830-f3f2b2bea8ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:10:44.224917 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a077d8-43a2-498a-9830-f3f2b2bea8ac service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:10:44.242546 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 666/1320] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:10:43 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:10:47.011477 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7571d6e-59ef-4873-a797-3df09385a99b None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:47.012006 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7571d6e-59ef-4873-a797-3df09385a99b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:47.012123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7571d6e-59ef-4873-a797-3df09385a99b None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:47.013448 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 655/1321] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:10:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:10:47.018798 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd110b5-3392-4ea5-9ede-b11b1ea8416d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:47.019270 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd110b5-3392-4ea5-9ede-b11b1ea8416d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:47.019367 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd110b5-3392-4ea5-9ede-b11b1ea8416d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:47.094201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ebd110b5-3392-4ea5-9ede-b11b1ea8416d None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:10:47.095351 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ebd110b5-3392-4ea5-9ede-b11b1ea8416d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:47.156501 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:10:47.199446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 667/1322] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:10:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:10:47.484061 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7571d6e-59ef-4873-a797-3df09385a99b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:10:47.485624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a7571d6e-59ef-4873-a797-3df09385a99b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:47.505735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:10:47.589291 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7571d6e-59ef-4873-a797-3df09385a99b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:10:47.662303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223c95ee-5f93-4534-91d6-0610ec6a9dde service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:10:47.663332 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223c95ee-5f93-4534-91d6-0610ec6a9dde service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:47.663610 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223c95ee-5f93-4534-91d6-0610ec6a9dde service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:47.663877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223c95ee-5f93-4534-91d6-0610ec6a9dde service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:47.665454 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-223c95ee-5f93-4534-91d6-0610ec6a9dde service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:47.735149 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:10:47.774152 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223c95ee-5f93-4534-91d6-0610ec6a9dde service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:10:47.775609 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223c95ee-5f93-4534-91d6-0610ec6a9dde service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:10:47.793109 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 656/1323] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:10:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:10:49.993841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa38b44-06eb-41a7-8ec6-49d8081b2a36 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:49.994389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa38b44-06eb-41a7-8ec6-49d8081b2a36 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:49.994389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa38b44-06eb-41a7-8ec6-49d8081b2a36 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:49.995930 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 668/1324] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:10:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:10:50.001368 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b547e6d6-3781-4cd4-8954-f4929e18ab94 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:50.001811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b547e6d6-3781-4cd4-8954-f4929e18ab94 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:50.001917 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b547e6d6-3781-4cd4-8954-f4929e18ab94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:50.067054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b547e6d6-3781-4cd4-8954-f4929e18ab94 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:10:50.068716 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b547e6d6-3781-4cd4-8954-f4929e18ab94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:50.174868 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 657/1325] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:10:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:10:50.474083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3615e5f-73a3-47ea-9ad8-d7e49bfb0aab None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:50.475091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3615e5f-73a3-47ea-9ad8-d7e49bfb0aab None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:50.475432 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3615e5f-73a3-47ea-9ad8-d7e49bfb0aab None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:50.476844 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 669/1326] 10.222.0.15 () {54 vars in 965 bytes} [Tue Sep 13 01:10:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:10:50.482761 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a590c9-1a87-442f-b5b0-f98e90627231 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:50.483130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a590c9-1a87-442f-b5b0-f98e90627231 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:50.483488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a590c9-1a87-442f-b5b0-f98e90627231 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:50.515119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:10:50.560424 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b4a590c9-1a87-442f-b5b0-f98e90627231 None None] MFA Rules not processed for user `09cd263f72484733adaaa0850ebfa609`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:10:50.561601 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b4a590c9-1a87-442f-b5b0-f98e90627231 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:50.636314 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 658/1327] 10.222.0.15 () {58 vars in 1062 bytes} [Tue Sep 13 01:10:50 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:10:50.645244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3615e5f-73a3-47ea-9ad8-d7e49bfb0aab None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:10:50.647426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d3615e5f-73a3-47ea-9ad8-d7e49bfb0aab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:50.752707 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3615e5f-73a3-47ea-9ad8-d7e49bfb0aab None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:10:50.827500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3d827b-c406-4d76-a062-0eff683cdb29 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:10:50.828100 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3d827b-c406-4d76-a062-0eff683cdb29 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:50.828373 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3d827b-c406-4d76-a062-0eff683cdb29 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:50.828643 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3d827b-c406-4d76-a062-0eff683cdb29 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:50.830198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1e3d827b-c406-4d76-a062-0eff683cdb29 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:50.847709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:10:50.941028 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3d827b-c406-4d76-a062-0eff683cdb29 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:10:50.942491 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3d827b-c406-4d76-a062-0eff683cdb29 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:10:50.961376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 670/1328] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:10:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:10:53.420678 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5319594c-5930-474a-a6ee-ac73da38ea67 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:53.421210 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5319594c-5930-474a-a6ee-ac73da38ea67 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:53.421210 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5319594c-5930-474a-a6ee-ac73da38ea67 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:53.422576 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 659/1329] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:10:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:10:53.427985 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e750e0-d03f-4589-b5d2-2c31ffb599ec None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:53.428387 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e750e0-d03f-4589-b5d2-2c31ffb599ec None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:53.428533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e750e0-d03f-4589-b5d2-2c31ffb599ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:53.498458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-63e750e0-d03f-4589-b5d2-2c31ffb599ec None None] MFA Rules not processed for user `1e80dcbdbef640078cfec6d97eabca95`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:10:53.499729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-63e750e0-d03f-4589-b5d2-2c31ffb599ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:53.568404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:10:53.601198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 671/1330] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:10:53 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:10:53.887698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5319594c-5930-474a-a6ee-ac73da38ea67 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:10:53.889220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5319594c-5930-474a-a6ee-ac73da38ea67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:53.993220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5319594c-5930-474a-a6ee-ac73da38ea67 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:10:54.017440 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:10:54.078674 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a438ce-c9ca-4e85-b562-c39c762054af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:10:54.079316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a438ce-c9ca-4e85-b562-c39c762054af service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:54.079590 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a438ce-c9ca-4e85-b562-c39c762054af service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:54.079857 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a438ce-c9ca-4e85-b562-c39c762054af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:54.081459 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-02a438ce-c9ca-4e85-b562-c39c762054af service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:54.191526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a438ce-c9ca-4e85-b562-c39c762054af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:10:54.193500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a438ce-c9ca-4e85-b562-c39c762054af service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:10:54.212242 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 660/1331] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:10:53 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:10:56.757910 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d07c1c8-6935-4f3f-8987-fb2f3cb1436e None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:56.758440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d07c1c8-6935-4f3f-8987-fb2f3cb1436e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:56.758440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d07c1c8-6935-4f3f-8987-fb2f3cb1436e None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:56.759823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 672/1332] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:10:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:10:56.765067 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67be96fd-b011-4460-af96-f4dba8b4e3d0 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:56.765505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67be96fd-b011-4460-af96-f4dba8b4e3d0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:56.765681 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67be96fd-b011-4460-af96-f4dba8b4e3d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:56.828588 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:10:56.831326 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-67be96fd-b011-4460-af96-f4dba8b4e3d0 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:10:56.833076 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-67be96fd-b011-4460-af96-f4dba8b4e3d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:56.927064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 661/1333] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:10:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:10:57.210497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d07c1c8-6935-4f3f-8987-fb2f3cb1436e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:10:57.212060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6d07c1c8-6935-4f3f-8987-fb2f3cb1436e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:57.313371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d07c1c8-6935-4f3f-8987-fb2f3cb1436e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:10:57.384797 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a469d6b-91dd-4be9-be0b-20daba50de6c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:10:57.385388 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a469d6b-91dd-4be9-be0b-20daba50de6c service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:57.385660 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a469d6b-91dd-4be9-be0b-20daba50de6c service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:57.385924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a469d6b-91dd-4be9-be0b-20daba50de6c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:57.387504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7a469d6b-91dd-4be9-be0b-20daba50de6c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:57.415587 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:10:57.497138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a469d6b-91dd-4be9-be0b-20daba50de6c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:10:57.498556 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a469d6b-91dd-4be9-be0b-20daba50de6c service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:10:57.517489 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 673/1334] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:10:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:10:59.689755 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca0293b-7e61-4c51-8e11-09a1ea5967ad None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:59.690319 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca0293b-7e61-4c51-8e11-09a1ea5967ad None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:59.690319 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca0293b-7e61-4c51-8e11-09a1ea5967ad None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:59.691618 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 662/1335] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:10:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:10:59.697408 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a1b590-9989-496e-af6f-56afae4a5f5e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:10:59.697811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a1b590-9989-496e-af6f-56afae4a5f5e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:10:59.697954 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a1b590-9989-496e-af6f-56afae4a5f5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:10:59.762286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-52a1b590-9989-496e-af6f-56afae4a5f5e None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:10:59.763492 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-52a1b590-9989-496e-af6f-56afae4a5f5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:10:59.847826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:10:59.873960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 674/1336] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:10:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:00.155619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca0293b-7e61-4c51-8e11-09a1ea5967ad None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:11:00.157157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0ca0293b-7e61-4c51-8e11-09a1ea5967ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:00.258292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca0293b-7e61-4c51-8e11-09a1ea5967ad None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:11:00.285261 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:00.328917 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f527894-b6e7-4d53-9380-2de4aa42a888 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:11:00.329514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f527894-b6e7-4d53-9380-2de4aa42a888 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:00.329786 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f527894-b6e7-4d53-9380-2de4aa42a888 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:00.330052 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f527894-b6e7-4d53-9380-2de4aa42a888 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:00.331633 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7f527894-b6e7-4d53-9380-2de4aa42a888 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:00.444260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f527894-b6e7-4d53-9380-2de4aa42a888 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:11:00.445690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f527894-b6e7-4d53-9380-2de4aa42a888 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:11:00.463404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 663/1337] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:11:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:11:02.900733 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae34bc0-ee86-407b-9a2b-960baa0b662f None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:02.901394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae34bc0-ee86-407b-9a2b-960baa0b662f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:02.901394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae34bc0-ee86-407b-9a2b-960baa0b662f None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:02.902616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 675/1338] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:11:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:02.907764 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f543c1-9d73-4323-b217-cd1a3ff43819 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:02.908164 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f543c1-9d73-4323-b217-cd1a3ff43819 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:02.908312 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f543c1-9d73-4323-b217-cd1a3ff43819 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:02.973292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c5f543c1-9d73-4323-b217-cd1a3ff43819 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:02.974481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c5f543c1-9d73-4323-b217-cd1a3ff43819 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:02.982772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:03.068823 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 664/1339] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:11:02 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:03.361365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae34bc0-ee86-407b-9a2b-960baa0b662f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:11:03.362934 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7ae34bc0-ee86-407b-9a2b-960baa0b662f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:03.468308 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae34bc0-ee86-407b-9a2b-960baa0b662f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:11:03.539692 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82c86a7-7a00-4417-87fa-8c0181575d81 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:11:03.540293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82c86a7-7a00-4417-87fa-8c0181575d81 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:03.540572 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82c86a7-7a00-4417-87fa-8c0181575d81 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:03.540890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82c86a7-7a00-4417-87fa-8c0181575d81 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:03.542420 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b82c86a7-7a00-4417-87fa-8c0181575d81 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:03.577371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:03.649487 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82c86a7-7a00-4417-87fa-8c0181575d81 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:11:03.650918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82c86a7-7a00-4417-87fa-8c0181575d81 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:11:03.668264 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 676/1340] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:11:03 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:11:07.746097 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86707a9b-d2f9-4261-88e4-dc6b7a2d8cbc None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:07.746642 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86707a9b-d2f9-4261-88e4-dc6b7a2d8cbc None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:07.746642 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86707a9b-d2f9-4261-88e4-dc6b7a2d8cbc None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:07.748023 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 665/1341] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:11:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:07.753669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c20f043-ebbe-4b94-9726-bb791d636cb2 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:07.754259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c20f043-ebbe-4b94-9726-bb791d636cb2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:07.754901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c20f043-ebbe-4b94-9726-bb791d636cb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:07.822165 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-0c20f043-ebbe-4b94-9726-bb791d636cb2 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:07.823310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0c20f043-ebbe-4b94-9726-bb791d636cb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:07.907172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:07.918827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 677/1342] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:11:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:08.200801 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86707a9b-d2f9-4261-88e4-dc6b7a2d8cbc None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:11:08.202343 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-86707a9b-d2f9-4261-88e4-dc6b7a2d8cbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:08.308351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86707a9b-d2f9-4261-88e4-dc6b7a2d8cbc None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:11:08.346880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:08.384208 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58c91c8-9e97-442c-8e15-a99320f55870 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:11:08.384815 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58c91c8-9e97-442c-8e15-a99320f55870 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:08.385093 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58c91c8-9e97-442c-8e15-a99320f55870 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:08.385364 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58c91c8-9e97-442c-8e15-a99320f55870 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:08.386972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f58c91c8-9e97-442c-8e15-a99320f55870 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:08.497477 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58c91c8-9e97-442c-8e15-a99320f55870 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:11:08.498935 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58c91c8-9e97-442c-8e15-a99320f55870 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:11:08.518150 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 666/1343] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:11:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:11:11.479264 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1c5470-ab9d-41f5-b916-ba40b72af9b7 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:11.479856 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1c5470-ab9d-41f5-b916-ba40b72af9b7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:11.479856 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1c5470-ab9d-41f5-b916-ba40b72af9b7 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:11.481157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 678/1344] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:11:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:11.486275 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2c4e7a-510c-4df2-b17f-15014048ae26 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:11.486729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2c4e7a-510c-4df2-b17f-15014048ae26 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:11.486841 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2c4e7a-510c-4df2-b17f-15014048ae26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:11.556615 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-4a2c4e7a-510c-4df2-b17f-15014048ae26 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:11.557823 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4a2c4e7a-510c-4df2-b17f-15014048ae26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:11.576422 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:11.659928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 667/1345] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:11:11 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:11.945729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1c5470-ab9d-41f5-b916-ba40b72af9b7 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:11:11.947365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0a1c5470-ab9d-41f5-b916-ba40b72af9b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:12.049180 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1c5470-ab9d-41f5-b916-ba40b72af9b7 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:11:12.126866 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76204285-f340-4752-ad92-961d9b0a39e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:11:12.127490 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76204285-f340-4752-ad92-961d9b0a39e4 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:12.127786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76204285-f340-4752-ad92-961d9b0a39e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:12.128051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76204285-f340-4752-ad92-961d9b0a39e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:12.130193 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-76204285-f340-4752-ad92-961d9b0a39e4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:12.172558 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:12.239100 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76204285-f340-4752-ad92-961d9b0a39e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:11:12.240525 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76204285-f340-4752-ad92-961d9b0a39e4 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:11:12.263199 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 679/1346] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:11:11 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:11:16.560147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b34d4df-c008-428c-911d-d2dd7c70eb0f None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:16.560672 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b34d4df-c008-428c-911d-d2dd7c70eb0f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:16.560672 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b34d4df-c008-428c-911d-d2dd7c70eb0f None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:16.562147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 668/1347] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:11:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:16.567898 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b487c89-df7b-448a-8a34-a0596179a566 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:16.568369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b487c89-df7b-448a-8a34-a0596179a566 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:16.568460 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b487c89-df7b-448a-8a34-a0596179a566 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:16.642356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9b487c89-df7b-448a-8a34-a0596179a566 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:16.643718 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9b487c89-df7b-448a-8a34-a0596179a566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:16.756703 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:16.759468 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 680/1348] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:11:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:17.037934 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b34d4df-c008-428c-911d-d2dd7c70eb0f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:11:17.039501 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0b34d4df-c008-428c-911d-d2dd7c70eb0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:17.141261 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b34d4df-c008-428c-911d-d2dd7c70eb0f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:11:17.196640 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:17.223142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca904de7-4171-486c-a812-64d018c1216a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:11:17.223752 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca904de7-4171-486c-a812-64d018c1216a service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:17.224033 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca904de7-4171-486c-a812-64d018c1216a service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:17.224303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca904de7-4171-486c-a812-64d018c1216a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:17.225909 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ca904de7-4171-486c-a812-64d018c1216a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:17.333265 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca904de7-4171-486c-a812-64d018c1216a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:11:17.334622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca904de7-4171-486c-a812-64d018c1216a service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:11:17.352748 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 669/1349] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:11:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:11:22.251133 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7ed2af-1ce3-4e90-b4d2-2147959ec7e9 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:22.252412 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7ed2af-1ce3-4e90-b4d2-2147959ec7e9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:22.252911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7ed2af-1ce3-4e90-b4d2-2147959ec7e9 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:22.255002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 681/1350] 10.222.0.15 () {54 vars in 921 bytes} [Tue Sep 13 01:11:22 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:22.260484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcd59b7-3f1a-44fe-98e7-23537b217e7c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:22.260911 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcd59b7-3f1a-44fe-98e7-23537b217e7c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:22.261268 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcd59b7-3f1a-44fe-98e7-23537b217e7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:22.354830 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-2bcd59b7-3f1a-44fe-98e7-23537b217e7c None None] MFA Rules not processed for user `8eaff1b843a145a49994dcf089f44315`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:22.356143 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2bcd59b7-3f1a-44fe-98e7-23537b217e7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:22.376209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:22.510041 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 670/1351] 10.222.0.15 () {58 vars in 1018 bytes} [Tue Sep 13 01:11:22 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:26.761394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2958c7c-a766-49c7-b39f-3ddad14fd5c8 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:26.762245 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2958c7c-a766-49c7-b39f-3ddad14fd5c8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:26.762512 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2958c7c-a766-49c7-b39f-3ddad14fd5c8 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:26.764107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 682/1352] 10.222.0.15 () {54 vars in 921 bytes} [Tue Sep 13 01:11:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:26.769959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9172b58-7b24-41f9-9530-07b772388475 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:26.770385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9172b58-7b24-41f9-9530-07b772388475 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:26.770497 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9172b58-7b24-41f9-9530-07b772388475 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:26.850836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-a9172b58-7b24-41f9-9530-07b772388475 None None] MFA Rules not processed for user `8eaff1b843a145a49994dcf089f44315`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:26.852065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a9172b58-7b24-41f9-9530-07b772388475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:26.925319 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:26.965470 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 671/1353] 10.222.0.15 () {58 vars in 1018 bytes} [Tue Sep 13 01:11:26 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:28.354409 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dac178-92ea-4d00-beae-5e48fa40d678 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:28.354973 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dac178-92ea-4d00-beae-5e48fa40d678 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:28.354973 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dac178-92ea-4d00-beae-5e48fa40d678 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:28.356315 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 683/1354] 10.222.0.15 () {54 vars in 921 bytes} [Tue Sep 13 01:11:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:28.361292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9d169c-85ae-4026-9813-0f7c03aad2d5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:28.361696 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9d169c-85ae-4026-9813-0f7c03aad2d5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:28.361845 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9d169c-85ae-4026-9813-0f7c03aad2d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:28.444182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ef9d169c-85ae-4026-9813-0f7c03aad2d5 None None] MFA Rules not processed for user `8eaff1b843a145a49994dcf089f44315`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:28.445423 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ef9d169c-85ae-4026-9813-0f7c03aad2d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:28.559372 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 672/1355] 10.222.0.15 () {58 vars in 1018 bytes} [Tue Sep 13 01:11:28 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:11:32.001881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de74ab15-44dc-4f27-aed6-8b894631e855 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:32.003019 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de74ab15-44dc-4f27-aed6-8b894631e855 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:32.003019 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de74ab15-44dc-4f27-aed6-8b894631e855 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:32.004474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 684/1356] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:11:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:32.009822 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947bef71-d34b-4818-a68f-82bc11f39254 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:32.010259 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947bef71-d34b-4818-a68f-82bc11f39254 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:32.010372 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947bef71-d34b-4818-a68f-82bc11f39254 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:32.047135 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:32.081555 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-947bef71-d34b-4818-a68f-82bc11f39254 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:32.082786 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-947bef71-d34b-4818-a68f-82bc11f39254 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:32.191118 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 673/1357] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:11:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:32.497979 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e6efac-2862-479b-8272-aa20ddcc2813 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:32.498562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e6efac-2862-479b-8272-aa20ddcc2813 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:32.499075 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e6efac-2862-479b-8272-aa20ddcc2813 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:32.501107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 685/1358] 10.222.0.15 () {54 vars in 962 bytes} [Tue Sep 13 01:11:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:32.513885 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef46a40-2a77-4da4-ac50-7b7a5eeaf7a7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:32.514230 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef46a40-2a77-4da4-ac50-7b7a5eeaf7a7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:32.514574 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef46a40-2a77-4da4-ac50-7b7a5eeaf7a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:32.604512 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-bef46a40-2a77-4da4-ac50-7b7a5eeaf7a7 None None] MFA Rules not processed for user `2f2da4d2d3604d16b58dc25b8c17c49a`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:32.605691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-bef46a40-2a77-4da4-ac50-7b7a5eeaf7a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:32.654823 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:32.727301 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 674/1359] 10.222.0.15 () {58 vars in 1059 bytes} [Tue Sep 13 01:11:32 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:32.738893 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e6efac-2862-479b-8272-aa20ddcc2813 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:11:32.742710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c7e6efac-2862-479b-8272-aa20ddcc2813 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:32.903688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e6efac-2862-479b-8272-aa20ddcc2813 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:11:32.980860 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:33.023039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b32cca-9d71-47eb-b1ac-b5f2917b8d22 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:11:33.023727 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b32cca-9d71-47eb-b1ac-b5f2917b8d22 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:33.024059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b32cca-9d71-47eb-b1ac-b5f2917b8d22 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:33.024399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b32cca-9d71-47eb-b1ac-b5f2917b8d22 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:33.026044 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a9b32cca-9d71-47eb-b1ac-b5f2917b8d22 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:33.129074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b32cca-9d71-47eb-b1ac-b5f2917b8d22 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:11:33.131132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b32cca-9d71-47eb-b1ac-b5f2917b8d22 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:11:33.153348 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 686/1360] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:11:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:11:35.561458 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0534946f-0f3e-4b9d-b6ef-9007833542ff None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:35.562071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0534946f-0f3e-4b9d-b6ef-9007833542ff None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:35.562071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0534946f-0f3e-4b9d-b6ef-9007833542ff None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:35.570840 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c2df46-c619-4c7e-91e5-d72059ed66d3 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:35.571552 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c2df46-c619-4c7e-91e5-d72059ed66d3 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:35.571552 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c2df46-c619-4c7e-91e5-d72059ed66d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:35.574255 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 675/1361] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:11:35 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:35.652042 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-87c2df46-c619-4c7e-91e5-d72059ed66d3 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:35.653341 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-87c2df46-c619-4c7e-91e5-d72059ed66d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:35.694440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:35.761586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 687/1362] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:11:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:36.047372 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0534946f-0f3e-4b9d-b6ef-9007833542ff None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:11:36.048894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0534946f-0f3e-4b9d-b6ef-9007833542ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:36.228120 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0534946f-0f3e-4b9d-b6ef-9007833542ff None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:11:36.240103 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:36.359949 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b62aed2-c729-4ca7-aed9-33319b0fc981 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:11:36.360561 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b62aed2-c729-4ca7-aed9-33319b0fc981 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:36.360837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b62aed2-c729-4ca7-aed9-33319b0fc981 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:36.361104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b62aed2-c729-4ca7-aed9-33319b0fc981 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:36.362706 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4b62aed2-c729-4ca7-aed9-33319b0fc981 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:36.457081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b62aed2-c729-4ca7-aed9-33319b0fc981 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:11:36.458713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b62aed2-c729-4ca7-aed9-33319b0fc981 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:11:36.477332 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 676/1363] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:11:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:11:38.977173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa91184-fdfd-44ce-8dd8-1ef01015abc9 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:38.977707 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa91184-fdfd-44ce-8dd8-1ef01015abc9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:38.977707 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa91184-fdfd-44ce-8dd8-1ef01015abc9 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:38.979083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 688/1364] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:11:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:38.984542 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6759dcbf-7fe2-4e72-9ef7-a419abbf3d72 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:38.984939 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6759dcbf-7fe2-4e72-9ef7-a419abbf3d72 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:38.985089 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6759dcbf-7fe2-4e72-9ef7-a419abbf3d72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:39.001444 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:39.055679 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6759dcbf-7fe2-4e72-9ef7-a419abbf3d72 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:39.056808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6759dcbf-7fe2-4e72-9ef7-a419abbf3d72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:39.161965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 677/1365] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:11:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:39.454631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721b0ad3-a811-4597-831c-554f457a7453 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:39.454944 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721b0ad3-a811-4597-831c-554f457a7453 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:39.455383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721b0ad3-a811-4597-831c-554f457a7453 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:39.456833 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 689/1366] 10.222.0.15 () {54 vars in 962 bytes} [Tue Sep 13 01:11:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:39.462000 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03ca70c-81f4-4960-b08b-c17bb9e6b502 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:39.462392 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03ca70c-81f4-4960-b08b-c17bb9e6b502 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:39.462768 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03ca70c-81f4-4960-b08b-c17bb9e6b502 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:39.541111 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e03ca70c-81f4-4960-b08b-c17bb9e6b502 None None] MFA Rules not processed for user `2f2da4d2d3604d16b58dc25b8c17c49a`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:39.542452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e03ca70c-81f4-4960-b08b-c17bb9e6b502 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:39.579288 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:39.665386 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 678/1367] 10.222.0.15 () {58 vars in 1059 bytes} [Tue Sep 13 01:11:39 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:39.673237 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721b0ad3-a811-4597-831c-554f457a7453 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:11:39.674959 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-721b0ad3-a811-4597-831c-554f457a7453 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:39.833833 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721b0ad3-a811-4597-831c-554f457a7453 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:11:39.856838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:39.965410 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c27222-f8fd-4217-b0ad-3ce1d27ec616 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:11:39.966125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c27222-f8fd-4217-b0ad-3ce1d27ec616 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:39.966528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c27222-f8fd-4217-b0ad-3ce1d27ec616 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:39.967103 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c27222-f8fd-4217-b0ad-3ce1d27ec616 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:39.969006 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d9c27222-f8fd-4217-b0ad-3ce1d27ec616 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:40.068944 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c27222-f8fd-4217-b0ad-3ce1d27ec616 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:11:40.070773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c27222-f8fd-4217-b0ad-3ce1d27ec616 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:11:40.092945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 690/1368] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:11:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:11:40.176481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6986de-00aa-43a4-8c64-dd3155ae14f3 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:40.177042 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6986de-00aa-43a4-8c64-dd3155ae14f3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:40.177042 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6986de-00aa-43a4-8c64-dd3155ae14f3 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:40.178545 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 679/1369] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:11:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:11:40.184316 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e3df6f-e527-4ef0-bcae-75aaa1fb26a4 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:40.184828 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e3df6f-e527-4ef0-bcae-75aaa1fb26a4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:40.185177 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e3df6f-e527-4ef0-bcae-75aaa1fb26a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:40.228280 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:40.274391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a0e3df6f-e527-4ef0-bcae-75aaa1fb26a4 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:40.275671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a0e3df6f-e527-4ef0-bcae-75aaa1fb26a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:40.392658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 691/1370] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:11:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:40.727458 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894f1457-fbbe-4903-8e0e-05eb899dae93 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:40.728009 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894f1457-fbbe-4903-8e0e-05eb899dae93 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:40.728546 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894f1457-fbbe-4903-8e0e-05eb899dae93 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:40.730966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 680/1371] 10.222.0.15 () {54 vars in 964 bytes} [Tue Sep 13 01:11:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:40.738026 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea52f327-c86c-44fb-a956-b1f72a534ea9 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:40.738377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea52f327-c86c-44fb-a956-b1f72a534ea9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:40.738649 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea52f327-c86c-44fb-a956-b1f72a534ea9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:40.823054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ea52f327-c86c-44fb-a956-b1f72a534ea9 None None] MFA Rules not processed for user `f90a5e57daa643eab94c01e21fcbc4c4`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:40.824622 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ea52f327-c86c-44fb-a956-b1f72a534ea9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:40.876585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:40.911567 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 692/1372] 10.222.0.15 () {58 vars in 1061 bytes} [Tue Sep 13 01:11:40 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:40.919555 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894f1457-fbbe-4903-8e0e-05eb899dae93 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:11:40.921135 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-894f1457-fbbe-4903-8e0e-05eb899dae93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:41.030455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894f1457-fbbe-4903-8e0e-05eb899dae93 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:11:41.078130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:41.122647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91e64c6-a8a6-4d67-b3fb-39a0bff29529 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:11:41.123504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91e64c6-a8a6-4d67-b3fb-39a0bff29529 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:41.124078 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91e64c6-a8a6-4d67-b3fb-39a0bff29529 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:41.124311 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91e64c6-a8a6-4d67-b3fb-39a0bff29529 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:41.126435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e91e64c6-a8a6-4d67-b3fb-39a0bff29529 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:41.270560 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91e64c6-a8a6-4d67-b3fb-39a0bff29529 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:11:41.272737 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e91e64c6-a8a6-4d67-b3fb-39a0bff29529 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:11:41.301271 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 681/1373] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:11:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:11:42.374088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6e02e1-40a8-4a16-8c71-cfd7ab3d162c None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:42.374678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6e02e1-40a8-4a16-8c71-cfd7ab3d162c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:42.374678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6e02e1-40a8-4a16-8c71-cfd7ab3d162c None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:42.376017 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 693/1374] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:11:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:42.381275 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c5ea58-aec2-45c0-9f57-860bd4728896 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:42.381734 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c5ea58-aec2-45c0-9f57-860bd4728896 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:42.381836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c5ea58-aec2-45c0-9f57-860bd4728896 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:42.454204 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-33c5ea58-aec2-45c0-9f57-860bd4728896 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:42.456054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-33c5ea58-aec2-45c0-9f57-860bd4728896 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:42.476667 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:42.556323 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 682/1375] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:11:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:42.850457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6e02e1-40a8-4a16-8c71-cfd7ab3d162c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:11:42.852062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0c6e02e1-40a8-4a16-8c71-cfd7ab3d162c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:43.006490 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6e02e1-40a8-4a16-8c71-cfd7ab3d162c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:11:43.060709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:43.126390 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf6de36-84cc-4538-bcd1-f4a9aa62afdd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:11:43.127054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf6de36-84cc-4538-bcd1-f4a9aa62afdd service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:43.127329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf6de36-84cc-4538-bcd1-f4a9aa62afdd service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:43.127596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf6de36-84cc-4538-bcd1-f4a9aa62afdd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:43.129170 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3cf6de36-84cc-4538-bcd1-f4a9aa62afdd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:43.225826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf6de36-84cc-4538-bcd1-f4a9aa62afdd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:11:43.227573 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf6de36-84cc-4538-bcd1-f4a9aa62afdd service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:11:43.246051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 694/1376] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:11:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:11:45.465914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fee9c7-fe61-4eef-af2d-eff7fc896a3b None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:45.466483 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fee9c7-fe61-4eef-af2d-eff7fc896a3b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:45.466483 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fee9c7-fe61-4eef-af2d-eff7fc896a3b None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:45.467984 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 683/1377] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:11:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:45.473229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b1dce-5a9b-438b-8017-45318fc7e906 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:45.473609 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b1dce-5a9b-438b-8017-45318fc7e906 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:45.473791 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b1dce-5a9b-438b-8017-45318fc7e906 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:45.541909 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-457b1dce-5a9b-438b-8017-45318fc7e906 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:45.543124 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-457b1dce-5a9b-438b-8017-45318fc7e906 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:45.551109 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:45.654284 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 695/1378] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:11:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:11:45.948316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fee9c7-fe61-4eef-af2d-eff7fc896a3b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:11:45.949861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c7fee9c7-fe61-4eef-af2d-eff7fc896a3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:46.105796 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fee9c7-fe61-4eef-af2d-eff7fc896a3b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:11:46.109442 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:46.224817 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74984454-fcc0-497f-adba-d556db023db9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:11:46.225437 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74984454-fcc0-497f-adba-d556db023db9 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:46.225712 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74984454-fcc0-497f-adba-d556db023db9 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:46.225971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74984454-fcc0-497f-adba-d556db023db9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:46.227581 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-74984454-fcc0-497f-adba-d556db023db9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:46.320430 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74984454-fcc0-497f-adba-d556db023db9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:11:46.322140 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74984454-fcc0-497f-adba-d556db023db9 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:11:46.342135 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 684/1379] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:11:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:11:48.579006 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae4ab88-1370-4cc7-9cf4-7279929167c0 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:48.579464 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae4ab88-1370-4cc7-9cf4-7279929167c0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:48.580149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae4ab88-1370-4cc7-9cf4-7279929167c0 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:48.581591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 696/1380] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:11:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:48.586998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bd3e49-8a84-4a94-92aa-3ee2e52ea4ae None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:48.587890 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bd3e49-8a84-4a94-92aa-3ee2e52ea4ae None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:48.588410 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bd3e49-8a84-4a94-92aa-3ee2e52ea4ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:48.602192 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:48.674115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-70bd3e49-8a84-4a94-92aa-3ee2e52ea4ae None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:48.675348 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-70bd3e49-8a84-4a94-92aa-3ee2e52ea4ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:48.801137 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 685/1381] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:11:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:49.091007 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae4ab88-1370-4cc7-9cf4-7279929167c0 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:11:49.092562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7ae4ab88-1370-4cc7-9cf4-7279929167c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:49.262778 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:49.275641 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae4ab88-1370-4cc7-9cf4-7279929167c0 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:11:49.396539 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192fd0f3-5836-4082-8dce-530fa5acf9d0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:11:49.397061 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192fd0f3-5836-4082-8dce-530fa5acf9d0 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:49.397335 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192fd0f3-5836-4082-8dce-530fa5acf9d0 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:49.397601 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192fd0f3-5836-4082-8dce-530fa5acf9d0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:49.399223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-192fd0f3-5836-4082-8dce-530fa5acf9d0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:49.501813 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192fd0f3-5836-4082-8dce-530fa5acf9d0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:11:49.503510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192fd0f3-5836-4082-8dce-530fa5acf9d0 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:11:49.513460 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:49.522487 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 697/1382] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:11:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:11:52.416308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8256048a-b1f3-427e-82ff-0496410e87cb None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:52.416748 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8256048a-b1f3-427e-82ff-0496410e87cb None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:52.417091 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8256048a-b1f3-427e-82ff-0496410e87cb None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:52.418843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 686/1383] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:11:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:52.425878 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c878828f-e33a-41a9-8e91-e113e3b0d364 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:52.426299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c878828f-e33a-41a9-8e91-e113e3b0d364 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:52.426415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c878828f-e33a-41a9-8e91-e113e3b0d364 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:52.494523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c878828f-e33a-41a9-8e91-e113e3b0d364 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:52.495869 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c878828f-e33a-41a9-8e91-e113e3b0d364 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:52.610500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 698/1384] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:11:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:52.896360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8256048a-b1f3-427e-82ff-0496410e87cb None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:11:52.898360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8256048a-b1f3-427e-82ff-0496410e87cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:52.998934 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:53.066624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8256048a-b1f3-427e-82ff-0496410e87cb None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:11:53.190108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c08137-131d-48bf-b276-8867f31581f8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:11:53.190656 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c08137-131d-48bf-b276-8867f31581f8 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:53.190958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c08137-131d-48bf-b276-8867f31581f8 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:53.191225 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c08137-131d-48bf-b276-8867f31581f8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:53.192818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d8c08137-131d-48bf-b276-8867f31581f8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:53.246216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:53.286553 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c08137-131d-48bf-b276-8867f31581f8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:11:53.288189 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c08137-131d-48bf-b276-8867f31581f8 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:11:53.306629 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 687/1385] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:11:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:11:55.469284 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7db037-23d4-4fcf-a111-bc98ba5da6e4 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:55.469796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7db037-23d4-4fcf-a111-bc98ba5da6e4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:55.469796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7db037-23d4-4fcf-a111-bc98ba5da6e4 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:55.471167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 699/1386] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:11:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:55.476372 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc97329-de46-4211-a766-0f3628b03776 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:55.476798 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc97329-de46-4211-a766-0f3628b03776 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:55.476933 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc97329-de46-4211-a766-0f3628b03776 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:55.542616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-fdc97329-de46-4211-a766-0f3628b03776 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:55.543866 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fdc97329-de46-4211-a766-0f3628b03776 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:55.639992 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 688/1387] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:11:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:55.939776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7db037-23d4-4fcf-a111-bc98ba5da6e4 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:11:55.941375 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1d7db037-23d4-4fcf-a111-bc98ba5da6e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:56.020400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:56.103666 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7db037-23d4-4fcf-a111-bc98ba5da6e4 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:11:56.218712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deefd8c8-c5c5-43ab-9a82-aa815749bd0b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:11:56.219290 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deefd8c8-c5c5-43ab-9a82-aa815749bd0b service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:56.220093 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deefd8c8-c5c5-43ab-9a82-aa815749bd0b service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:56.220363 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deefd8c8-c5c5-43ab-9a82-aa815749bd0b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:56.221918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-deefd8c8-c5c5-43ab-9a82-aa815749bd0b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:56.261641 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:56.313978 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deefd8c8-c5c5-43ab-9a82-aa815749bd0b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:11:56.315592 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deefd8c8-c5c5-43ab-9a82-aa815749bd0b service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:11:56.333743 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 700/1388] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:11:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:11:56.626644 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fe4162-18ed-44de-9581-5c12b989b620 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:56.627244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fe4162-18ed-44de-9581-5c12b989b620 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:56.627244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fe4162-18ed-44de-9581-5c12b989b620 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:56.628651 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 689/1389] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:11:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:56.634021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40ce57f-e5f8-4a90-a91a-3bff0b1356ea None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:56.634458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40ce57f-e5f8-4a90-a91a-3bff0b1356ea None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:56.634553 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40ce57f-e5f8-4a90-a91a-3bff0b1356ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:56.701656 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b40ce57f-e5f8-4a90-a91a-3bff0b1356ea None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:56.702875 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b40ce57f-e5f8-4a90-a91a-3bff0b1356ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:56.805364 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 701/1390] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:11:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:57.114457 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d073c9f5-fe22-4986-81e1-688c005d194c None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:57.114769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d073c9f5-fe22-4986-81e1-688c005d194c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:57.115092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d073c9f5-fe22-4986-81e1-688c005d194c None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:57.116469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 690/1391] 10.222.0.15 () {54 vars in 966 bytes} [Tue Sep 13 01:11:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:57.121870 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7c7cc6-14b2-4457-94a3-bc250c34a370 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:57.122225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7c7cc6-14b2-4457-94a3-bc250c34a370 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:57.122568 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7c7cc6-14b2-4457-94a3-bc250c34a370 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:57.139454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:57.209217 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-cf7c7cc6-14b2-4457-94a3-bc250c34a370 None None] MFA Rules not processed for user `eef98bcf2699461d9a7f3c3bffe82f34`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:57.211658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-cf7c7cc6-14b2-4457-94a3-bc250c34a370 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:57.292805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 702/1392] 10.222.0.15 () {58 vars in 1063 bytes} [Tue Sep 13 01:11:57 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:57.305308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d073c9f5-fe22-4986-81e1-688c005d194c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:11:57.306956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d073c9f5-fe22-4986-81e1-688c005d194c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:57.338552 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:57.416628 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d073c9f5-fe22-4986-81e1-688c005d194c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:11:57.490094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4d25e0-a04a-4f59-8aa7-6292a44e461b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eef98bcf2699461d9a7f3c3bffe82f34', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:11:57.490701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4d25e0-a04a-4f59-8aa7-6292a44e461b service glance] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:57.491020 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4d25e0-a04a-4f59-8aa7-6292a44e461b service glance] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:57.491295 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4d25e0-a04a-4f59-8aa7-6292a44e461b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:57.492903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1e4d25e0-a04a-4f59-8aa7-6292a44e461b service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:57.573181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:57.609335 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4d25e0-a04a-4f59-8aa7-6292a44e461b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:11:57.610702 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4d25e0-a04a-4f59-8aa7-6292a44e461b service glance] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:11:57.629094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 691/1393] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:11:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:11:58.428374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f53515b-dc8b-4c25-95ee-2e3373d229bd None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:58.428942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f53515b-dc8b-4c25-95ee-2e3373d229bd None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:58.428942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f53515b-dc8b-4c25-95ee-2e3373d229bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:58.501921 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9f53515b-dc8b-4c25-95ee-2e3373d229bd None None] MFA Rules not processed for user `e28a0ba45389417ba8ae92376dba87ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:58.503178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9f53515b-dc8b-4c25-95ee-2e3373d229bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:58.558691 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:58.592562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 703/1394] 10.222.0.15 () {58 vars in 1009 bytes} [Tue Sep 13 01:11:58 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:58.642087 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2db2096-e344-49f8-8a83-fedf9fd3f451 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:58.642664 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2db2096-e344-49f8-8a83-fedf9fd3f451 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:58.642664 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2db2096-e344-49f8-8a83-fedf9fd3f451 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:58.644099 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 692/1395] 10.222.0.15 () {54 vars in 943 bytes} [Tue Sep 13 01:11:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:11:58.651557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b0a398-11b8-463f-a90e-af48980393a5 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:58.651930 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b0a398-11b8-463f-a90e-af48980393a5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:58.652275 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b0a398-11b8-463f-a90e-af48980393a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:58.733158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-17b0a398-11b8-463f-a90e-af48980393a5 None None] MFA Rules not processed for user `e0d7d8ed4ede4ddca192fce0a497689d`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:58.734361 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-17b0a398-11b8-463f-a90e-af48980393a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:58.761538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de9be70-d929-4bf7-8a8b-ba7ecc72d01f None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:58.761813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de9be70-d929-4bf7-8a8b-ba7ecc72d01f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:58.762097 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de9be70-d929-4bf7-8a8b-ba7ecc72d01f None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:58.763525 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 693/1396] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:11:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:11:58.769153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d0e58b-d2a0-47f4-b9d5-14b71437bd37 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:58.769441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d0e58b-d2a0-47f4-b9d5-14b71437bd37 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:58.769721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d0e58b-d2a0-47f4-b9d5-14b71437bd37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:58.837019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f4d0e58b-d2a0-47f4-b9d5-14b71437bd37 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:11:58.838238 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f4d0e58b-d2a0-47f4-b9d5-14b71437bd37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:58.860693 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 704/1397] 10.222.0.15 () {58 vars in 1040 bytes} [Tue Sep 13 01:11:58 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:11:58.869960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b0a398-11b8-463f-a90e-af48980393a5 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:11:58.879810 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b0a398-11b8-463f-a90e-af48980393a5 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:11:58.906173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:58.938768 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 694/1398] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:11:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:11:58.992590 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e7162f-c94d-4652-a8b9-19f8c9685119 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d7d8ed4ede4ddca192fce0a497689d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:11:58.993195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e7162f-c94d-4652-a8b9-19f8c9685119 service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:58.993471 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e7162f-c94d-4652-a8b9-19f8c9685119 service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:58.993741 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e7162f-c94d-4652-a8b9-19f8c9685119 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:58.995400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-91e7162f-c94d-4652-a8b9-19f8c9685119 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:59.083828 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e7162f-c94d-4652-a8b9-19f8c9685119 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:11:59.085436 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e7162f-c94d-4652-a8b9-19f8c9685119 service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:11:59.093406 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 705/1399] 10.222.0.15 () {60 vars in 1384 bytes} [Tue Sep 13 01:11:58 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 13 01:11:59.233682 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d0e58b-d2a0-47f4-b9d5-14b71437bd37 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:11:59.234987 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f4d0e58b-d2a0-47f4-b9d5-14b71437bd37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:59.282244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:59.367403 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d0e58b-d2a0-47f4-b9d5-14b71437bd37 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:11:59.499861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33332d54-884f-4cde-b40f-c41eb720032e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:11:59.500468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33332d54-884f-4cde-b40f-c41eb720032e service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:11:59.501316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33332d54-884f-4cde-b40f-c41eb720032e service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:11:59.501588 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33332d54-884f-4cde-b40f-c41eb720032e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:11:59.503250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-33332d54-884f-4cde-b40f-c41eb720032e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:11:59.542623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:11:59.603087 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33332d54-884f-4cde-b40f-c41eb720032e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:11:59.604682 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33332d54-884f-4cde-b40f-c41eb720032e service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:11:59.624530 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 695/1400] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:11:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:12:02.274715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e635aea-9163-44af-a548-6472d98fe09f None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:02.275364 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e635aea-9163-44af-a548-6472d98fe09f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:02.275364 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e635aea-9163-44af-a548-6472d98fe09f None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:02.276919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 706/1401] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:12:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:12:02.282074 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7979d8-b3c9-4c87-9555-c19fb163bc8c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:02.282510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7979d8-b3c9-4c87-9555-c19fb163bc8c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:02.282622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7979d8-b3c9-4c87-9555-c19fb163bc8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:02.356493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-0c7979d8-b3c9-4c87-9555-c19fb163bc8c None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:02.357869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0c7979d8-b3c9-4c87-9555-c19fb163bc8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:02.472794 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 696/1402] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:12:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:12:02.764200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e635aea-9163-44af-a548-6472d98fe09f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:12:02.765730 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3e635aea-9163-44af-a548-6472d98fe09f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:02.833936 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:02.928690 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e635aea-9163-44af-a548-6472d98fe09f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:12:03.052230 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4894d542-3642-423a-acb4-00c6c51028b4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:12:03.052752 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4894d542-3642-423a-acb4-00c6c51028b4 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:03.053029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4894d542-3642-423a-acb4-00c6c51028b4 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:03.053297 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4894d542-3642-423a-acb4-00c6c51028b4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:03.055180 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4894d542-3642-423a-acb4-00c6c51028b4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:03.078013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:03.163567 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4894d542-3642-423a-acb4-00c6c51028b4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:12:03.165154 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4894d542-3642-423a-acb4-00c6c51028b4 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:12:03.183969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 707/1403] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:12:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:12:05.410060 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e0ae00-040d-483a-92af-e9a85a3c55b8 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:05.410607 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e0ae00-040d-483a-92af-e9a85a3c55b8 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:05.410607 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e0ae00-040d-483a-92af-e9a85a3c55b8 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:05.415890 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 697/1404] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:12:05 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:12:05.417236 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d520397-b166-4155-b657-f0ed24fdc333 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:05.417671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d520397-b166-4155-b657-f0ed24fdc333 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:05.417765 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d520397-b166-4155-b657-f0ed24fdc333 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:05.494057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7d520397-b166-4155-b657-f0ed24fdc333 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:05.495303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7d520397-b166-4155-b657-f0ed24fdc333 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:05.597703 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:05.605235 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 708/1405] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:12:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:12:05.895073 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e0ae00-040d-483a-92af-e9a85a3c55b8 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:12:05.896628 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-71e0ae00-040d-483a-92af-e9a85a3c55b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:05.909588 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:06.059670 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e0ae00-040d-483a-92af-e9a85a3c55b8 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:12:06.159643 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:06.198370 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d5b369-1b4c-4a3b-8cb4-f5e55805d64f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:12:06.199042 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d5b369-1b4c-4a3b-8cb4-f5e55805d64f service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:06.199322 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d5b369-1b4c-4a3b-8cb4-f5e55805d64f service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:06.199593 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d5b369-1b4c-4a3b-8cb4-f5e55805d64f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:06.201214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e5d5b369-1b4c-4a3b-8cb4-f5e55805d64f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:06.306368 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d5b369-1b4c-4a3b-8cb4-f5e55805d64f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:12:06.308147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d5b369-1b4c-4a3b-8cb4-f5e55805d64f service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:12:06.327573 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 698/1406] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:12:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:12:08.540019 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ef8d3f-5629-437e-b145-928f306c6abc None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:08.540621 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ef8d3f-5629-437e-b145-928f306c6abc None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:08.540730 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ef8d3f-5629-437e-b145-928f306c6abc None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:08.542015 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 709/1407] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:12:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:12:08.547469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658a2658-f3b1-4c6d-8807-0693e5cd65f5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:08.547899 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658a2658-f3b1-4c6d-8807-0693e5cd65f5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:08.548003 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658a2658-f3b1-4c6d-8807-0693e5cd65f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:08.617684 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-658a2658-f3b1-4c6d-8807-0693e5cd65f5 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:08.619285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-658a2658-f3b1-4c6d-8807-0693e5cd65f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:08.669812 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:08.725669 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 699/1408] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:12:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:12:09.011920 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ef8d3f-5629-437e-b145-928f306c6abc None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:12:09.013447 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-97ef8d3f-5629-437e-b145-928f306c6abc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:09.174593 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ef8d3f-5629-437e-b145-928f306c6abc None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:12:09.250093 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:09.300942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e55064-4388-4429-81cf-3403bf6e9e77 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:12:09.301667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e55064-4388-4429-81cf-3403bf6e9e77 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:09.302010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e55064-4388-4429-81cf-3403bf6e9e77 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:09.302352 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e55064-4388-4429-81cf-3403bf6e9e77 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:09.304061 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e4e55064-4388-4429-81cf-3403bf6e9e77 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:09.411098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e55064-4388-4429-81cf-3403bf6e9e77 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:12:09.415201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e55064-4388-4429-81cf-3403bf6e9e77 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:12:09.436421 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 710/1409] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:12:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:12:18.724089 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=23 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:19.434156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=64 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:21.941094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2087aa-5a23-4fa8-8d33-92c41f229e1c None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:21.941641 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2087aa-5a23-4fa8-8d33-92c41f229e1c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:21.941641 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2087aa-5a23-4fa8-8d33-92c41f229e1c None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:21.942995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 700/1410] 10.222.0.15 () {54 vars in 918 bytes} [Tue Sep 13 01:12:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:12:21.947963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446f723f-4a31-4042-b62c-2be7e71081b5 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:21.948393 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446f723f-4a31-4042-b62c-2be7e71081b5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:21.948523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446f723f-4a31-4042-b62c-2be7e71081b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:22.024869 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-446f723f-4a31-4042-b62c-2be7e71081b5 None None] MFA Rules not processed for user `8eaff1b843a145a49994dcf089f44315`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:22.026066 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-446f723f-4a31-4042-b62c-2be7e71081b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:22.131056 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 711/1411] 10.222.0.15 () {58 vars in 1015 bytes} [Tue Sep 13 01:12:21 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:12:22.145567 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3aa4a3c-39c2-4e72-b86d-1dc19a8db086 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:22.145851 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3aa4a3c-39c2-4e72-b86d-1dc19a8db086 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:22.146122 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3aa4a3c-39c2-4e72-b86d-1dc19a8db086 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:22.147475 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 701/1412] 10.222.0.15 () {54 vars in 918 bytes} [Tue Sep 13 01:12:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:12:22.152106 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50c40e8-63f3-4f7e-9f77-586800b5da4c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:22.152391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50c40e8-63f3-4f7e-9f77-586800b5da4c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:22.152668 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50c40e8-63f3-4f7e-9f77-586800b5da4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:22.224318 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:22.227662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e50c40e8-63f3-4f7e-9f77-586800b5da4c None None] MFA Rules not processed for user `2f2da4d2d3604d16b58dc25b8c17c49a`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:22.229528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e50c40e8-63f3-4f7e-9f77-586800b5da4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:22.348305 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 712/1413] 10.222.0.15 () {58 vars in 1015 bytes} [Tue Sep 13 01:12:22 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:12:24.902522 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ac17df-a6e1-4a76-b22f-450232ee55e7 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:24.903134 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ac17df-a6e1-4a76-b22f-450232ee55e7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:24.903134 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ac17df-a6e1-4a76-b22f-450232ee55e7 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:24.904454 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 702/1414] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 01:12:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:12:24.910189 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd21255-691d-4ece-8c57-d3c749ff5153 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:24.910602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd21255-691d-4ece-8c57-d3c749ff5153 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:24.910743 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd21255-691d-4ece-8c57-d3c749ff5153 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:24.976829 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ddd21255-691d-4ece-8c57-d3c749ff5153 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:24.978027 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ddd21255-691d-4ece-8c57-d3c749ff5153 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:25.032970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:25.075313 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 713/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 01:12:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:12:25.361365 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ac17df-a6e1-4a76-b22f-450232ee55e7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:12:25.362991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d6ac17df-a6e1-4a76-b22f-450232ee55e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:25.513998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ac17df-a6e1-4a76-b22f-450232ee55e7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:12:25.585470 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:25.625032 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84443293-7d3c-45a0-8a21-234d3034350a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:12:25.625570 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84443293-7d3c-45a0-8a21-234d3034350a service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:25.625849 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84443293-7d3c-45a0-8a21-234d3034350a service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:25.626120 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84443293-7d3c-45a0-8a21-234d3034350a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:25.627733 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-84443293-7d3c-45a0-8a21-234d3034350a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:25.715795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84443293-7d3c-45a0-8a21-234d3034350a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:12:25.717382 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84443293-7d3c-45a0-8a21-234d3034350a service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:12:25.735201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 703/1416] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:12:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:12:26.892209 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9951ae-c6ad-4798-8bb6-05bc0380c1d5 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:26.892778 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9951ae-c6ad-4798-8bb6-05bc0380c1d5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:26.893170 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9951ae-c6ad-4798-8bb6-05bc0380c1d5 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:26.895810 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 714/1417] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 01:12:26 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:12:26.902178 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2311e24-2ba5-4a53-b024-d004174097aa None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:26.902635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2311e24-2ba5-4a53-b024-d004174097aa None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:26.902742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2311e24-2ba5-4a53-b024-d004174097aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:26.978632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-a2311e24-2ba5-4a53-b024-d004174097aa None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:26.979903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a2311e24-2ba5-4a53-b024-d004174097aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:27.006290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:27.074298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 704/1418] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 01:12:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:12:27.224785 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9951ae-c6ad-4798-8bb6-05bc0380c1d5 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:12:27.226292 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7c9951ae-c6ad-4798-8bb6-05bc0380c1d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:27.368352 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9951ae-c6ad-4798-8bb6-05bc0380c1d5 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:12:27.409219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:27.480441 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b665cda5-3fce-4831-a656-d07086f99188 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:12:27.480945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b665cda5-3fce-4831-a656-d07086f99188 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:27.481223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b665cda5-3fce-4831-a656-d07086f99188 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:27.481494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b665cda5-3fce-4831-a656-d07086f99188 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:27.483114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b665cda5-3fce-4831-a656-d07086f99188 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:27.571582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b665cda5-3fce-4831-a656-d07086f99188 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:12:27.573144 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b665cda5-3fce-4831-a656-d07086f99188 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:12:27.591340 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 715/1419] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:12:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:12:28.718130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d0f514-fc8d-45e1-9f19-ed4d9c2b0e46 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:28.718639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d0f514-fc8d-45e1-9f19-ed4d9c2b0e46 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:28.718639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d0f514-fc8d-45e1-9f19-ed4d9c2b0e46 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:28.719989 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 705/1420] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 01:12:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:12:28.725986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2217f2-0f9f-4684-a20d-27800a4fefbe None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:28.726390 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2217f2-0f9f-4684-a20d-27800a4fefbe None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:28.726536 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2217f2-0f9f-4684-a20d-27800a4fefbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:28.779695 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:28.801665 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8e2217f2-0f9f-4684-a20d-27800a4fefbe None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:28.802846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8e2217f2-0f9f-4684-a20d-27800a4fefbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:28.899372 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 716/1421] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 01:12:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:12:29.049923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d0f514-fc8d-45e1-9f19-ed4d9c2b0e46 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:12:29.051589 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a9d0f514-fc8d-45e1-9f19-ed4d9c2b0e46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:29.194472 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d0f514-fc8d-45e1-9f19-ed4d9c2b0e46 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:12:29.206330 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:29.306504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8b1026-b614-40ac-8f8e-cc5500829ba1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:12:29.307059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8b1026-b614-40ac-8f8e-cc5500829ba1 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:29.307335 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8b1026-b614-40ac-8f8e-cc5500829ba1 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:29.307600 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8b1026-b614-40ac-8f8e-cc5500829ba1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:29.309170 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-dd8b1026-b614-40ac-8f8e-cc5500829ba1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:29.397168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8b1026-b614-40ac-8f8e-cc5500829ba1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:12:29.398757 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8b1026-b614-40ac-8f8e-cc5500829ba1 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:12:29.417397 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 706/1422] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:12:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:12:30.532672 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fc5a3d-1f9a-478e-9411-c709677abbbf None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:30.533208 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fc5a3d-1f9a-478e-9411-c709677abbbf None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:30.533208 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fc5a3d-1f9a-478e-9411-c709677abbbf None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:30.534603 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 717/1423] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 01:12:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:12:30.540950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd722813-4353-4b96-8aae-d7c2bdf39630 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:30.541387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd722813-4353-4b96-8aae-d7c2bdf39630 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:30.541495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd722813-4353-4b96-8aae-d7c2bdf39630 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:30.561633 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:30.617344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-bd722813-4353-4b96-8aae-d7c2bdf39630 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:30.618567 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-bd722813-4353-4b96-8aae-d7c2bdf39630 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:30.713762 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 707/1424] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 01:12:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:12:30.862824 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fc5a3d-1f9a-478e-9411-c709677abbbf None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:12:30.864373 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b2fc5a3d-1f9a-478e-9411-c709677abbbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:30.973465 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:31.015861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fc5a3d-1f9a-478e-9411-c709677abbbf None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:12:31.127891 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edbf4af-7780-4d44-87ae-ebb8b1d04466 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:12:31.128489 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edbf4af-7780-4d44-87ae-ebb8b1d04466 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:31.128763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edbf4af-7780-4d44-87ae-ebb8b1d04466 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:31.129031 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edbf4af-7780-4d44-87ae-ebb8b1d04466 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:31.130589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5edbf4af-7780-4d44-87ae-ebb8b1d04466 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:31.207950 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:31.221913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edbf4af-7780-4d44-87ae-ebb8b1d04466 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:12:31.223582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edbf4af-7780-4d44-87ae-ebb8b1d04466 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:12:31.240990 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 718/1425] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:12:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:12:32.360416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a81ab13-3d2a-4262-9764-a8a728763155 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:32.360928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a81ab13-3d2a-4262-9764-a8a728763155 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:32.361036 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a81ab13-3d2a-4262-9764-a8a728763155 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:32.362350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 708/1426] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 01:12:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:12:32.368434 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b024ffdb-0df9-4dc8-b731-1502ae3f11e7 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:32.368845 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b024ffdb-0df9-4dc8-b731-1502ae3f11e7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:32.369003 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b024ffdb-0df9-4dc8-b731-1502ae3f11e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:32.444917 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b024ffdb-0df9-4dc8-b731-1502ae3f11e7 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:32.446097 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b024ffdb-0df9-4dc8-b731-1502ae3f11e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:32.543631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 719/1427] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 01:12:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:12:32.693597 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a81ab13-3d2a-4262-9764-a8a728763155 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:12:32.695064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4a81ab13-3d2a-4262-9764-a8a728763155 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:32.776926 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:32.838244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a81ab13-3d2a-4262-9764-a8a728763155 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:12:32.954415 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71387713-581f-4dc6-aa2d-6521028cbc25 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:12:32.955016 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71387713-581f-4dc6-aa2d-6521028cbc25 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:32.955299 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71387713-581f-4dc6-aa2d-6521028cbc25 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:32.955570 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71387713-581f-4dc6-aa2d-6521028cbc25 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:32.957835 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-71387713-581f-4dc6-aa2d-6521028cbc25 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:33.008614 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:33.050125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71387713-581f-4dc6-aa2d-6521028cbc25 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:12:33.051731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71387713-581f-4dc6-aa2d-6521028cbc25 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:12:33.069495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 709/1428] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:12:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:12:34.128240 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039cc5a3-0654-4e49-b049-cae6d4a8fa4f None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:34.128808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039cc5a3-0654-4e49-b049-cae6d4a8fa4f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:34.128808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039cc5a3-0654-4e49-b049-cae6d4a8fa4f None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:34.130121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 720/1429] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 01:12:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:12:34.136158 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161f8ae6-891b-4e0f-8e5c-09d80bdaa3da None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:34.136594 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161f8ae6-891b-4e0f-8e5c-09d80bdaa3da None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:34.136707 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161f8ae6-891b-4e0f-8e5c-09d80bdaa3da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:34.226173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-161f8ae6-891b-4e0f-8e5c-09d80bdaa3da None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:34.227432 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-161f8ae6-891b-4e0f-8e5c-09d80bdaa3da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:34.323193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 710/1430] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 01:12:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:12:34.473344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039cc5a3-0654-4e49-b049-cae6d4a8fa4f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:12:34.474970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-039cc5a3-0654-4e49-b049-cae6d4a8fa4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:34.525250 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:34.622181 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039cc5a3-0654-4e49-b049-cae6d4a8fa4f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:12:34.736274 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01d9de0-b00a-4a93-ae7a-e2fb2f0808b9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:12:34.736886 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01d9de0-b00a-4a93-ae7a-e2fb2f0808b9 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:34.737168 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01d9de0-b00a-4a93-ae7a-e2fb2f0808b9 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:34.737441 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01d9de0-b00a-4a93-ae7a-e2fb2f0808b9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:34.739072 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f01d9de0-b00a-4a93-ae7a-e2fb2f0808b9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:34.746156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:34.834344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01d9de0-b00a-4a93-ae7a-e2fb2f0808b9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:12:34.835965 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01d9de0-b00a-4a93-ae7a-e2fb2f0808b9 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:12:34.853622 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 721/1431] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:12:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:12:35.894131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbd87b6-ca18-4448-98ed-8c833cf649ca None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:35.894705 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbd87b6-ca18-4448-98ed-8c833cf649ca None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:35.894705 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbd87b6-ca18-4448-98ed-8c833cf649ca None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:35.896078 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 711/1432] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 01:12:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:12:35.902224 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4a215-8f2c-46d4-a276-1a3a8d5e71dd None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:35.902651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4a215-8f2c-46d4-a276-1a3a8d5e71dd None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:35.902760 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4a215-8f2c-46d4-a276-1a3a8d5e71dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:35.985073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-79e4a215-8f2c-46d4-a276-1a3a8d5e71dd None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:35.986270 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-79e4a215-8f2c-46d4-a276-1a3a8d5e71dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:36.053039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:36.082322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 722/1433] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 01:12:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:12:36.233865 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbd87b6-ca18-4448-98ed-8c833cf649ca None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:12:36.235415 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2fbd87b6-ca18-4448-98ed-8c833cf649ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:36.253344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:36.380128 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbd87b6-ca18-4448-98ed-8c833cf649ca None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:12:36.468532 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:36.495861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bbaeca-6498-45a9-ba23-4237b7089d57 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:12:36.496408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bbaeca-6498-45a9-ba23-4237b7089d57 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:36.496684 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bbaeca-6498-45a9-ba23-4237b7089d57 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:36.496954 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bbaeca-6498-45a9-ba23-4237b7089d57 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:36.498559 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-00bbaeca-6498-45a9-ba23-4237b7089d57 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:36.590842 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bbaeca-6498-45a9-ba23-4237b7089d57 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:12:36.593036 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bbaeca-6498-45a9-ba23-4237b7089d57 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:12:36.611558 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 712/1434] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:12:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:12:37.659419 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8f0e6c-2f60-41ba-9de4-f5aee4e5bd8a None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:37.659995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8f0e6c-2f60-41ba-9de4-f5aee4e5bd8a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:37.659995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8f0e6c-2f60-41ba-9de4-f5aee4e5bd8a None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:37.661371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 723/1435] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 01:12:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:12:37.667352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578939b5-723c-4e6a-a7ba-7ef452b31c60 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:37.667787 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578939b5-723c-4e6a-a7ba-7ef452b31c60 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:37.667897 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578939b5-723c-4e6a-a7ba-7ef452b31c60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:37.742373 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-578939b5-723c-4e6a-a7ba-7ef452b31c60 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:37.743591 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-578939b5-723c-4e6a-a7ba-7ef452b31c60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:37.783781 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:37.838324 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 713/1436] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 01:12:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:12:37.988101 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8f0e6c-2f60-41ba-9de4-f5aee4e5bd8a None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:12:37.989627 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ce8f0e6c-2f60-41ba-9de4-f5aee4e5bd8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:38.143662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8f0e6c-2f60-41ba-9de4-f5aee4e5bd8a None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:12:38.197216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:38.256513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c963d96b-affe-444d-94cc-ee40d3ee8d7c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:12:38.257057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c963d96b-affe-444d-94cc-ee40d3ee8d7c service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:38.257338 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c963d96b-affe-444d-94cc-ee40d3ee8d7c service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:38.257611 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c963d96b-affe-444d-94cc-ee40d3ee8d7c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:38.259232 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c963d96b-affe-444d-94cc-ee40d3ee8d7c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:38.347710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c963d96b-affe-444d-94cc-ee40d3ee8d7c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:12:38.349299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c963d96b-affe-444d-94cc-ee40d3ee8d7c service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:12:38.367506 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 724/1437] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:12:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:12:39.414618 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac50889-262f-43a6-acb7-51462ac6dd51 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:39.415032 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac50889-262f-43a6-acb7-51462ac6dd51 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:39.415200 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac50889-262f-43a6-acb7-51462ac6dd51 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:39.416547 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 714/1438] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 01:12:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:12:39.423375 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf67c49c-ad81-434f-aa95-51cf672dcc60 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:39.423815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf67c49c-ad81-434f-aa95-51cf672dcc60 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:39.424020 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf67c49c-ad81-434f-aa95-51cf672dcc60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:39.499748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-bf67c49c-ad81-434f-aa95-51cf672dcc60 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:39.501815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-bf67c49c-ad81-434f-aa95-51cf672dcc60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:39.502957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:39.599609 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 725/1439] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 01:12:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:12:39.748160 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac50889-262f-43a6-acb7-51462ac6dd51 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:12:39.750225 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cac50889-262f-43a6-acb7-51462ac6dd51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:39.895401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac50889-262f-43a6-acb7-51462ac6dd51 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:12:39.922415 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:40.007004 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf368f92-5c59-4984-baac-ae02d4bb8314 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:12:40.007536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf368f92-5c59-4984-baac-ae02d4bb8314 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:40.007810 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf368f92-5c59-4984-baac-ae02d4bb8314 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:40.008078 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf368f92-5c59-4984-baac-ae02d4bb8314 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:40.009664 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-bf368f92-5c59-4984-baac-ae02d4bb8314 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:40.097085 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf368f92-5c59-4984-baac-ae02d4bb8314 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:12:40.098679 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf368f92-5c59-4984-baac-ae02d4bb8314 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:12:40.116720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 715/1440] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:12:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:12:42.174520 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf6d7e4-78a8-42e2-af79-9228d1aac482 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:42.175092 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf6d7e4-78a8-42e2-af79-9228d1aac482 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:42.175092 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf6d7e4-78a8-42e2-af79-9228d1aac482 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:42.176445 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 726/1441] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 01:12:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:12:42.181938 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1dd9bc-e17e-4846-af2f-e7afeefc5fe5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:42.182343 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1dd9bc-e17e-4846-af2f-e7afeefc5fe5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:42.182516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1dd9bc-e17e-4846-af2f-e7afeefc5fe5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:42.224009 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:42.255064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8c1dd9bc-e17e-4846-af2f-e7afeefc5fe5 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:42.256278 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8c1dd9bc-e17e-4846-af2f-e7afeefc5fe5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:42.350512 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 716/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 01:12:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:12:42.635684 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf6d7e4-78a8-42e2-af79-9228d1aac482 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:12:42.637222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-dbf6d7e4-78a8-42e2-af79-9228d1aac482 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:42.741007 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf6d7e4-78a8-42e2-af79-9228d1aac482 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:12:42.774186 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:42.816654 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16de0af-6c05-46eb-b926-1d76151c0a94 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:12:42.817259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16de0af-6c05-46eb-b926-1d76151c0a94 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:42.817534 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16de0af-6c05-46eb-b926-1d76151c0a94 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:42.817805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16de0af-6c05-46eb-b926-1d76151c0a94 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:42.819410 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a16de0af-6c05-46eb-b926-1d76151c0a94 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:42.926518 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a16de0af-6c05-46eb-b926-1d76151c0a94 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:12:42.927960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a16de0af-6c05-46eb-b926-1d76151c0a94 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:12:42.946421 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 727/1443] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:12:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:12:45.217206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa8f59b-6249-40f1-803e-df497555841e None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:45.217735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa8f59b-6249-40f1-803e-df497555841e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:45.217735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa8f59b-6249-40f1-803e-df497555841e None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:45.219594 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 717/1444] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 01:12:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:12:45.224805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e92a21-be05-47e6-8fc5-0b8a1d835c9c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:45.225235 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e92a21-be05-47e6-8fc5-0b8a1d835c9c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:45.225360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e92a21-be05-47e6-8fc5-0b8a1d835c9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:45.291943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-44e92a21-be05-47e6-8fc5-0b8a1d835c9c None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:45.293143 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-44e92a21-be05-47e6-8fc5-0b8a1d835c9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:45.311838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:45.397554 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 728/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 01:12:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:12:45.677682 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa8f59b-6249-40f1-803e-df497555841e None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:12:45.679267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5aa8f59b-6249-40f1-803e-df497555841e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:45.781884 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa8f59b-6249-40f1-803e-df497555841e None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:12:45.794592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:45.857075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02fc6a59-5232-4d60-a838-d2d8cf36747e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:12:45.857672 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fc6a59-5232-4d60-a838-d2d8cf36747e service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:45.857953 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fc6a59-5232-4d60-a838-d2d8cf36747e service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:45.858216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fc6a59-5232-4d60-a838-d2d8cf36747e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:45.860404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-02fc6a59-5232-4d60-a838-d2d8cf36747e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:45.966115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02fc6a59-5232-4d60-a838-d2d8cf36747e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:12:45.967499 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02fc6a59-5232-4d60-a838-d2d8cf36747e service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:12:45.985943 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 718/1446] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:12:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:12:48.477331 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a003ef41-4cd0-4fba-b51e-ef549838f6f1 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:48.477755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a003ef41-4cd0-4fba-b51e-ef549838f6f1 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:48.477883 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a003ef41-4cd0-4fba-b51e-ef549838f6f1 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:48.479284 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 729/1447] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 01:12:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:12:48.484477 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59f1e08-48fd-40f1-a2da-8799f30cdee3 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:48.485507 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59f1e08-48fd-40f1-a2da-8799f30cdee3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:48.485617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59f1e08-48fd-40f1-a2da-8799f30cdee3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:48.532297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:48.551110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c59f1e08-48fd-40f1-a2da-8799f30cdee3 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:48.552319 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c59f1e08-48fd-40f1-a2da-8799f30cdee3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:48.651031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 719/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 01:12:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:12:48.936317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a003ef41-4cd0-4fba-b51e-ef549838f6f1 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:12:48.937907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a003ef41-4cd0-4fba-b51e-ef549838f6f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:49.039831 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a003ef41-4cd0-4fba-b51e-ef549838f6f1 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:12:49.079624 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:49.113085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd51c6de-d424-4f04-b66b-e8bf1eec3945 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:12:49.113687 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd51c6de-d424-4f04-b66b-e8bf1eec3945 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:49.113964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd51c6de-d424-4f04-b66b-e8bf1eec3945 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:49.114235 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd51c6de-d424-4f04-b66b-e8bf1eec3945 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:49.115843 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-fd51c6de-d424-4f04-b66b-e8bf1eec3945 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:49.225047 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd51c6de-d424-4f04-b66b-e8bf1eec3945 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:12:49.226471 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd51c6de-d424-4f04-b66b-e8bf1eec3945 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:12:49.244716 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 730/1449] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:12:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:12:51.496472 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5992f53-07dc-4e12-838d-8ee39dbaa468 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:51.496993 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5992f53-07dc-4e12-838d-8ee39dbaa468 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:51.497123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5992f53-07dc-4e12-838d-8ee39dbaa468 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:51.498404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 720/1450] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 01:12:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:12:51.503727 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ea97f3-296a-4f10-a36b-f9667c40b374 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:51.504146 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ea97f3-296a-4f10-a36b-f9667c40b374 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:51.504278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ea97f3-296a-4f10-a36b-f9667c40b374 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:51.572092 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-00ea97f3-296a-4f10-a36b-f9667c40b374 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:51.573284 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-00ea97f3-296a-4f10-a36b-f9667c40b374 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:51.593351 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:51.669651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 731/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 01:12:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:12:51.954669 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5992f53-07dc-4e12-838d-8ee39dbaa468 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:12:51.956264 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f5992f53-07dc-4e12-838d-8ee39dbaa468 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:52.058326 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5992f53-07dc-4e12-838d-8ee39dbaa468 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:12:52.079306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:52.132249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b955f2-ac2c-40a7-b5f7-3869c7459f91 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:12:52.132853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b955f2-ac2c-40a7-b5f7-3869c7459f91 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:52.133130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b955f2-ac2c-40a7-b5f7-3869c7459f91 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:52.133399 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b955f2-ac2c-40a7-b5f7-3869c7459f91 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:52.135052 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-30b955f2-ac2c-40a7-b5f7-3869c7459f91 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:52.245070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b955f2-ac2c-40a7-b5f7-3869c7459f91 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:12:52.246497 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b955f2-ac2c-40a7-b5f7-3869c7459f91 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:12:52.264113 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 721/1452] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:12:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:12:54.446670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044810f9-e1ba-4085-b580-e768642611dd None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:54.447181 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044810f9-e1ba-4085-b580-e768642611dd None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:54.447292 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044810f9-e1ba-4085-b580-e768642611dd None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:54.448589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 732/1453] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 01:12:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:12:54.453818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875e753b-3540-4948-a6bd-17809968f9c5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:54.454268 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875e753b-3540-4948-a6bd-17809968f9c5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:54.454391 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875e753b-3540-4948-a6bd-17809968f9c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:54.517709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:54.520524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-875e753b-3540-4948-a6bd-17809968f9c5 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:54.522306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-875e753b-3540-4948-a6bd-17809968f9c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:54.627467 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 722/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 01:12:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:12:54.909582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044810f9-e1ba-4085-b580-e768642611dd None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:12:54.911225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-044810f9-e1ba-4085-b580-e768642611dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:55.015661 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044810f9-e1ba-4085-b580-e768642611dd None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:12:55.065732 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:55.089897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e841e533-1a8e-4ddc-b4d5-9e732b9ce957 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:12:55.090498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e841e533-1a8e-4ddc-b4d5-9e732b9ce957 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:55.090777 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e841e533-1a8e-4ddc-b4d5-9e732b9ce957 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:55.091075 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e841e533-1a8e-4ddc-b4d5-9e732b9ce957 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:55.092653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e841e533-1a8e-4ddc-b4d5-9e732b9ce957 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:55.199252 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e841e533-1a8e-4ddc-b4d5-9e732b9ce957 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:12:55.200597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e841e533-1a8e-4ddc-b4d5-9e732b9ce957 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:12:55.218443 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 733/1455] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:12:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:12:57.395133 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e020dcf-8e78-4f0a-b266-09ed0509c67c None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:57.395675 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e020dcf-8e78-4f0a-b266-09ed0509c67c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:57.395675 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e020dcf-8e78-4f0a-b266-09ed0509c67c None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:57.397004 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 723/1456] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 01:12:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:12:57.402342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b95d683-5da2-4828-be3b-fda7d7c9730e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:57.402775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b95d683-5da2-4828-be3b-fda7d7c9730e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:57.402903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b95d683-5da2-4828-be3b-fda7d7c9730e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:57.468956 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-1b95d683-5da2-4828-be3b-fda7d7c9730e None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:12:57.470938 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1b95d683-5da2-4828-be3b-fda7d7c9730e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:57.499524 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:57.567041 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 734/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 01:12:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:12:57.849765 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e020dcf-8e78-4f0a-b266-09ed0509c67c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:12:57.851343 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9e020dcf-8e78-4f0a-b266-09ed0509c67c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:57.954963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e020dcf-8e78-4f0a-b266-09ed0509c67c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:12:57.982756 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:12:58.028736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81258c2c-98a7-4cd1-b416-9b3184593ca5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:12:58.029352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81258c2c-98a7-4cd1-b416-9b3184593ca5 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:12:58.029615 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81258c2c-98a7-4cd1-b416-9b3184593ca5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:12:58.029885 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81258c2c-98a7-4cd1-b416-9b3184593ca5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:12:58.031502 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-81258c2c-98a7-4cd1-b416-9b3184593ca5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:12:58.141569 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81258c2c-98a7-4cd1-b416-9b3184593ca5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:12:58.143042 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81258c2c-98a7-4cd1-b416-9b3184593ca5 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:12:58.160605 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 724/1458] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:12:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:13:00.377413 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa4c93a-8036-4fa0-99c0-19ac1555ba84 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:00.377993 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa4c93a-8036-4fa0-99c0-19ac1555ba84 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:00.377993 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa4c93a-8036-4fa0-99c0-19ac1555ba84 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:00.379341 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 735/1459] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 01:13:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:13:00.385622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0af821b-0588-46a3-9003-97eb7c35eb7c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:00.386045 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0af821b-0588-46a3-9003-97eb7c35eb7c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:00.386158 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0af821b-0588-46a3-9003-97eb7c35eb7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:00.452540 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d0af821b-0588-46a3-9003-97eb7c35eb7c None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:13:00.453732 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d0af821b-0588-46a3-9003-97eb7c35eb7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:00.461967 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:00.550291 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 725/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 01:13:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:13:00.835453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa4c93a-8036-4fa0-99c0-19ac1555ba84 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:13:00.837069 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ffa4c93a-8036-4fa0-99c0-19ac1555ba84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:01.054442 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa4c93a-8036-4fa0-99c0-19ac1555ba84 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:13:01.109282 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:01.128386 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7321fd-4a72-4a90-9658-2241051be0fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:13:01.128982 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7321fd-4a72-4a90-9658-2241051be0fb service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:01.129254 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7321fd-4a72-4a90-9658-2241051be0fb service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:01.129518 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7321fd-4a72-4a90-9658-2241051be0fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:01.131117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7c7321fd-4a72-4a90-9658-2241051be0fb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:01.237316 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7321fd-4a72-4a90-9658-2241051be0fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:13:01.238729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7321fd-4a72-4a90-9658-2241051be0fb service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:13:01.256268 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 736/1461] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:13:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:13:03.449720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29725304-162e-461e-8f2b-c0798dbaa60d None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:03.450253 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29725304-162e-461e-8f2b-c0798dbaa60d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:03.450253 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29725304-162e-461e-8f2b-c0798dbaa60d None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:03.451622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 726/1462] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 01:13:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:13:03.457458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc772f81-8702-488b-9ca9-bbd23b1b6464 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:03.457865 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc772f81-8702-488b-9ca9-bbd23b1b6464 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:03.458010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc772f81-8702-488b-9ca9-bbd23b1b6464 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:03.523335 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-fc772f81-8702-488b-9ca9-bbd23b1b6464 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:13:03.524537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-fc772f81-8702-488b-9ca9-bbd23b1b6464 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:03.562356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:03.619614 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 737/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 01:13:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:13:03.912547 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29725304-162e-461e-8f2b-c0798dbaa60d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:13:03.914098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-29725304-162e-461e-8f2b-c0798dbaa60d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:04.023368 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29725304-162e-461e-8f2b-c0798dbaa60d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:13:04.060997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:04.097476 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486d1876-91be-4264-bc30-bf227bacc17f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:13:04.098094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486d1876-91be-4264-bc30-bf227bacc17f service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:04.098373 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486d1876-91be-4264-bc30-bf227bacc17f service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:04.098642 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486d1876-91be-4264-bc30-bf227bacc17f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:04.100265 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-486d1876-91be-4264-bc30-bf227bacc17f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:04.209661 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486d1876-91be-4264-bc30-bf227bacc17f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:13:04.211053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486d1876-91be-4264-bc30-bf227bacc17f service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:13:04.229062 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 727/1464] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:13:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:13:06.403586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a25c13-6311-4446-8c5a-f0333be5e6d3 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:06.404176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a25c13-6311-4446-8c5a-f0333be5e6d3 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:06.404176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a25c13-6311-4446-8c5a-f0333be5e6d3 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:06.405684 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 738/1465] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 01:13:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:13:06.411435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86982d4-de40-4f7b-9c24-93ecca71f7cb None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:06.411850 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86982d4-de40-4f7b-9c24-93ecca71f7cb None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:06.411979 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86982d4-de40-4f7b-9c24-93ecca71f7cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:06.477725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c86982d4-de40-4f7b-9c24-93ecca71f7cb None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:13:06.478977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c86982d4-de40-4f7b-9c24-93ecca71f7cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:06.497406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:06.575425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 728/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 01:13:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:13:06.859671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a25c13-6311-4446-8c5a-f0333be5e6d3 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:13:06.861730 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-98a25c13-6311-4446-8c5a-f0333be5e6d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:06.964565 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a25c13-6311-4446-8c5a-f0333be5e6d3 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:13:07.029372 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:07.036106 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c564f2e-b55f-4542-bd89-66906240267b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:13:07.036702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c564f2e-b55f-4542-bd89-66906240267b service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:07.036977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c564f2e-b55f-4542-bd89-66906240267b service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:07.037244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c564f2e-b55f-4542-bd89-66906240267b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:07.038812 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8c564f2e-b55f-4542-bd89-66906240267b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:07.147541 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c564f2e-b55f-4542-bd89-66906240267b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:13:07.148899 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c564f2e-b55f-4542-bd89-66906240267b service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:13:07.167292 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 739/1467] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:13:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:13:09.298067 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11893381-b087-4b52-8ef1-04bde669caaa None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:09.298641 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11893381-b087-4b52-8ef1-04bde669caaa None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:09.298641 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11893381-b087-4b52-8ef1-04bde669caaa None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:09.300030 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 729/1468] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 01:13:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:13:09.305217 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a088a50a-c827-42b1-ba7c-c8cb0faa9d0e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:09.305912 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a088a50a-c827-42b1-ba7c-c8cb0faa9d0e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:09.306018 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a088a50a-c827-42b1-ba7c-c8cb0faa9d0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:09.370508 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a088a50a-c827-42b1-ba7c-c8cb0faa9d0e None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:13:09.371708 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a088a50a-c827-42b1-ba7c-c8cb0faa9d0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:09.419439 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:09.467491 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 740/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 01:13:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:13:09.745461 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11893381-b087-4b52-8ef1-04bde669caaa None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:13:09.747068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-11893381-b087-4b52-8ef1-04bde669caaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:09.848536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11893381-b087-4b52-8ef1-04bde669caaa None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:13:09.898955 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:09.921028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a9c7e3d-e80d-4bbc-afa2-9ccf1a91d80b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:13:09.921627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9c7e3d-e80d-4bbc-afa2-9ccf1a91d80b service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:09.921903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9c7e3d-e80d-4bbc-afa2-9ccf1a91d80b service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:09.922180 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9c7e3d-e80d-4bbc-afa2-9ccf1a91d80b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:09.924276 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1a9c7e3d-e80d-4bbc-afa2-9ccf1a91d80b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:10.030675 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a9c7e3d-e80d-4bbc-afa2-9ccf1a91d80b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:13:10.032050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a9c7e3d-e80d-4bbc-afa2-9ccf1a91d80b service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:13:10.049951 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 730/1470] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:13:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:13:19.465953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=22 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:20.047296 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=66 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:29.881841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a17f40e-fd72-495f-a41d-bec33c05ff58 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:29.882452 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a17f40e-fd72-495f-a41d-bec33c05ff58 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:29.882452 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a17f40e-fd72-495f-a41d-bec33c05ff58 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:29.885116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 741/1471] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:13:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:13:29.890662 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20fc628-180d-401f-8dac-1158e27a16fc None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:29.891125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20fc628-180d-401f-8dac-1158e27a16fc None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:29.891230 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20fc628-180d-401f-8dac-1158e27a16fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:30.074301 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d20fc628-180d-401f-8dac-1158e27a16fc None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:13:30.075540 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d20fc628-180d-401f-8dac-1158e27a16fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:30.170253 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 731/1472] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:13:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:13:30.465707 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e70961-2147-4476-9e35-49dcd84a645f None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:30.466014 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e70961-2147-4476-9e35-49dcd84a645f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:30.466291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e70961-2147-4476-9e35-49dcd84a645f None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:30.467638 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 742/1473] 10.222.0.15 () {54 vars in 966 bytes} [Tue Sep 13 01:13:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:13:30.473220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb920350-c46f-4f77-b853-2812035dedea None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:30.473509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb920350-c46f-4f77-b853-2812035dedea None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:30.473789 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb920350-c46f-4f77-b853-2812035dedea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:30.548404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-cb920350-c46f-4f77-b853-2812035dedea None None] MFA Rules not processed for user `eef98bcf2699461d9a7f3c3bffe82f34`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:13:30.549596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cb920350-c46f-4f77-b853-2812035dedea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:30.569717 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:30.624129 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 732/1474] 10.222.0.15 () {58 vars in 1063 bytes} [Tue Sep 13 01:13:30 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:13:30.632637 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e70961-2147-4476-9e35-49dcd84a645f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:13:30.634165 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-59e70961-2147-4476-9e35-49dcd84a645f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:30.737956 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e70961-2147-4476-9e35-49dcd84a645f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:13:30.810000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceeea66d-bda2-4c53-a350-439b2a93bb10 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eef98bcf2699461d9a7f3c3bffe82f34', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:13:30.810597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceeea66d-bda2-4c53-a350-439b2a93bb10 service glance] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:30.810893 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceeea66d-bda2-4c53-a350-439b2a93bb10 service glance] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:30.811171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceeea66d-bda2-4c53-a350-439b2a93bb10 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:30.812736 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ceeea66d-bda2-4c53-a350-439b2a93bb10 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:30.866205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:30.920577 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceeea66d-bda2-4c53-a350-439b2a93bb10 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:13:30.922005 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceeea66d-bda2-4c53-a350-439b2a93bb10 service glance] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:13:30.940844 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 743/1475] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:13:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:13:33.134711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e734916-9ef0-4057-872c-631b39cc203d None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:33.135347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e734916-9ef0-4057-872c-631b39cc203d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:33.135347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e734916-9ef0-4057-872c-631b39cc203d None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:33.136691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 733/1476] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:13:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:13:33.141806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606ef743-798d-49fe-9a27-f17abdf64a26 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:33.142271 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606ef743-798d-49fe-9a27-f17abdf64a26 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:33.142376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606ef743-798d-49fe-9a27-f17abdf64a26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:33.209075 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-606ef743-798d-49fe-9a27-f17abdf64a26 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:13:33.210287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-606ef743-798d-49fe-9a27-f17abdf64a26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:33.305798 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 744/1477] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:13:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:13:33.584719 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e734916-9ef0-4057-872c-631b39cc203d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:13:33.586609 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8e734916-9ef0-4057-872c-631b39cc203d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:33.708928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e734916-9ef0-4057-872c-631b39cc203d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:13:33.756902 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:33.802100 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b964724-c95d-4ed8-862d-bc2ab862ddfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:13:33.802706 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b964724-c95d-4ed8-862d-bc2ab862ddfb admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:33.803007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b964724-c95d-4ed8-862d-bc2ab862ddfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:33.803282 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b964724-c95d-4ed8-862d-bc2ab862ddfb admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:33.803981 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b964724-c95d-4ed8-862d-bc2ab862ddfb admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:13:33.805772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: WARNING py.warnings [None req-1b964724-c95d-4ed8-862d-bc2ab862ddfb admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 01:13:33.805772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: warnings.warn(msg) Sep 13 01:13:33.805772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: Sep 13 01:13:33.807183 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b964724-c95d-4ed8-862d-bc2ab862ddfb admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:13:33.830293 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 734/1478] 10.222.0.15 () {56 vars in 1108 bytes} [Tue Sep 13 01:13:33 2022] GET /identity/v3/projects => generated 2553 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:13:35.994681 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264e75e4-f45f-49ba-85af-22d1f8a1b53a None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:35.995285 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264e75e4-f45f-49ba-85af-22d1f8a1b53a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:35.995285 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264e75e4-f45f-49ba-85af-22d1f8a1b53a None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:35.996591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 745/1479] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:13:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:13:36.002285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe98bf9e-6df7-4a52-a965-6a177d411461 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:36.002729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe98bf9e-6df7-4a52-a965-6a177d411461 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:36.002828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe98bf9e-6df7-4a52-a965-6a177d411461 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:36.069554 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-fe98bf9e-6df7-4a52-a965-6a177d411461 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:13:36.070771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fe98bf9e-6df7-4a52-a965-6a177d411461 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:36.166568 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 735/1480] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:13:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:13:36.461868 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264e75e4-f45f-49ba-85af-22d1f8a1b53a None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:13:36.463453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-264e75e4-f45f-49ba-85af-22d1f8a1b53a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:36.473714 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:36.609314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264e75e4-f45f-49ba-85af-22d1f8a1b53a None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:13:36.690298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:36.722755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5c1e93-5159-45da-9bb6-57d0f3ab55c9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:13:36.723380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5c1e93-5159-45da-9bb6-57d0f3ab55c9 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:36.723654 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5c1e93-5159-45da-9bb6-57d0f3ab55c9 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:36.723921 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5c1e93-5159-45da-9bb6-57d0f3ab55c9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:36.725471 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ca5c1e93-5159-45da-9bb6-57d0f3ab55c9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:36.813227 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5c1e93-5159-45da-9bb6-57d0f3ab55c9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:13:36.814788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5c1e93-5159-45da-9bb6-57d0f3ab55c9 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:13:36.833074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 746/1481] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:13:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:13:39.156062 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da9af04-ddca-4301-a5da-1fc1d798277f None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:39.156683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da9af04-ddca-4301-a5da-1fc1d798277f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:39.156683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da9af04-ddca-4301-a5da-1fc1d798277f None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:39.158570 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 736/1482] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:13:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:13:39.163843 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871aebab-8785-4aac-85cb-c33ba3b735c4 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:39.164249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871aebab-8785-4aac-85cb-c33ba3b735c4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:39.164369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871aebab-8785-4aac-85cb-c33ba3b735c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:39.229761 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-871aebab-8785-4aac-85cb-c33ba3b735c4 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:13:39.230977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-871aebab-8785-4aac-85cb-c33ba3b735c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:39.275706 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:39.325369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 747/1483] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:13:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:13:39.625403 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da9af04-ddca-4301-a5da-1fc1d798277f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:13:39.627505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9da9af04-ddca-4301-a5da-1fc1d798277f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:39.651471 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:39.730483 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da9af04-ddca-4301-a5da-1fc1d798277f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:13:39.802599 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6429e87c-a3bd-4f77-b2f3-988d692ca539 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:13:39.803222 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6429e87c-a3bd-4f77-b2f3-988d692ca539 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:39.803508 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6429e87c-a3bd-4f77-b2f3-988d692ca539 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:39.803779 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6429e87c-a3bd-4f77-b2f3-988d692ca539 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:39.805354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6429e87c-a3bd-4f77-b2f3-988d692ca539 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:39.882295 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:39.916572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6429e87c-a3bd-4f77-b2f3-988d692ca539 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:13:39.918001 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6429e87c-a3bd-4f77-b2f3-988d692ca539 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:13:39.936445 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 737/1484] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:13:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:13:42.120962 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcc37c3-09bc-4588-a090-a9254add3c9a None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:42.121510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcc37c3-09bc-4588-a090-a9254add3c9a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:42.121510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcc37c3-09bc-4588-a090-a9254add3c9a None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:42.122844 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 748/1485] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:13:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:13:42.128788 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ae8214-c98c-484d-97b0-5ddf79bf0228 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:42.129194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ae8214-c98c-484d-97b0-5ddf79bf0228 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:42.129330 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ae8214-c98c-484d-97b0-5ddf79bf0228 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:42.193687 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-84ae8214-c98c-484d-97b0-5ddf79bf0228 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:13:42.194960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-84ae8214-c98c-484d-97b0-5ddf79bf0228 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:42.288655 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 738/1486] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:13:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:13:42.575820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcc37c3-09bc-4588-a090-a9254add3c9a None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:13:42.577345 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7fcc37c3-09bc-4588-a090-a9254add3c9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:42.683462 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcc37c3-09bc-4588-a090-a9254add3c9a None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:13:42.755249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:42.756675 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15ee116-b429-42bf-a380-06e94f578ba6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:13:42.757288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15ee116-b429-42bf-a380-06e94f578ba6 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:42.757736 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15ee116-b429-42bf-a380-06e94f578ba6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:42.758163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15ee116-b429-42bf-a380-06e94f578ba6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:42.759944 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d15ee116-b429-42bf-a380-06e94f578ba6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:42.867322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15ee116-b429-42bf-a380-06e94f578ba6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:13:42.868734 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15ee116-b429-42bf-a380-06e94f578ba6 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:13:42.886258 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 749/1487] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:13:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:13:45.154125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f52f03-2e1b-46ef-9fce-c61ab798ff96 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:45.154651 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f52f03-2e1b-46ef-9fce-c61ab798ff96 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:45.154651 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f52f03-2e1b-46ef-9fce-c61ab798ff96 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:45.155964 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 739/1488] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:13:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:13:45.161755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f54a26f-a788-4938-8f82-c64c09d7196d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:45.162168 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f54a26f-a788-4938-8f82-c64c09d7196d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:45.162287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f54a26f-a788-4938-8f82-c64c09d7196d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:45.227357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8f54a26f-a788-4938-8f82-c64c09d7196d None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:13:45.228542 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8f54a26f-a788-4938-8f82-c64c09d7196d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:45.279379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:45.322898 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 750/1489] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:13:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:13:45.614201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f52f03-2e1b-46ef-9fce-c61ab798ff96 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:13:45.615835 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-91f52f03-2e1b-46ef-9fce-c61ab798ff96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:45.650715 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:45.717286 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f52f03-2e1b-46ef-9fce-c61ab798ff96 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:13:45.790619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebf6803-571a-40e1-966f-10b711c2e8c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:13:45.791248 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebf6803-571a-40e1-966f-10b711c2e8c4 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:45.791521 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebf6803-571a-40e1-966f-10b711c2e8c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:45.791790 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebf6803-571a-40e1-966f-10b711c2e8c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:45.793354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7ebf6803-571a-40e1-966f-10b711c2e8c4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:45.870908 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:45.899514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebf6803-571a-40e1-966f-10b711c2e8c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:13:45.900932 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebf6803-571a-40e1-966f-10b711c2e8c4 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:13:45.920013 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 740/1490] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:13:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:13:48.208291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950fb3b7-a29f-443d-a454-d8759267b035 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:48.208881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950fb3b7-a29f-443d-a454-d8759267b035 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:48.208881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950fb3b7-a29f-443d-a454-d8759267b035 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:48.210171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 751/1491] 10.222.0.15 () {54 vars in 927 bytes} [Tue Sep 13 01:13:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:13:48.215408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac9465e-c7ad-430e-b2e5-51f3dade5812 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:48.215853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac9465e-c7ad-430e-b2e5-51f3dade5812 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:48.215958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac9465e-c7ad-430e-b2e5-51f3dade5812 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:48.281284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9ac9465e-c7ad-430e-b2e5-51f3dade5812 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:13:48.282510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9ac9465e-c7ad-430e-b2e5-51f3dade5812 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:48.377280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 741/1492] 10.222.0.15 () {58 vars in 1024 bytes} [Tue Sep 13 01:13:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:13:48.665729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950fb3b7-a29f-443d-a454-d8759267b035 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:13:48.667278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-950fb3b7-a29f-443d-a454-d8759267b035 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:48.770970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950fb3b7-a29f-443d-a454-d8759267b035 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:13:48.847775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633dca28-3482-4088-8e85-d31d1dda0a26 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:13:48.848718 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633dca28-3482-4088-8e85-d31d1dda0a26 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:13:48.849173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633dca28-3482-4088-8e85-d31d1dda0a26 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:13:48.849618 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633dca28-3482-4088-8e85-d31d1dda0a26 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:13:48.851929 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-633dca28-3482-4088-8e85-d31d1dda0a26 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:13:48.860888 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:48.961984 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633dca28-3482-4088-8e85-d31d1dda0a26 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:13:48.963447 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633dca28-3482-4088-8e85-d31d1dda0a26 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:13:48.981355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 752/1493] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:13:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:13:58.375308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=82 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:13:58.979197 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=50 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:15:09.193879 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9422c7d-909a-45bf-aa44-800cfcdd00ee None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:15:09.194477 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9422c7d-909a-45bf-aa44-800cfcdd00ee None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:15:09.194477 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9422c7d-909a-45bf-aa44-800cfcdd00ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:15:09.268610 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b9422c7d-909a-45bf-aa44-800cfcdd00ee None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:15:09.269797 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b9422c7d-909a-45bf-aa44-800cfcdd00ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:15:09.362657 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 742/1494] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:15:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:15:09.370675 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9917948d-0574-4d71-9b50-a4dcf3aed1b7 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:15:09.371192 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9917948d-0574-4d71-9b50-a4dcf3aed1b7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:15:09.371294 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9917948d-0574-4d71-9b50-a4dcf3aed1b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:15:09.449755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9917948d-0574-4d71-9b50-a4dcf3aed1b7 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:15:09.450949 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9917948d-0574-4d71-9b50-a4dcf3aed1b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:15:09.545159 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 753/1495] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:15:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:15:09.549797 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9422c7d-909a-45bf-aa44-800cfcdd00ee None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:15:09.551033 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b9422c7d-909a-45bf-aa44-800cfcdd00ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:15:09.562433 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9422c7d-909a-45bf-aa44-800cfcdd00ee None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:15:09.626147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:15:09.659856 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06bc5b4-e294-472f-9e53-704b6b9072f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:15:09.660456 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06bc5b4-e294-472f-9e53-704b6b9072f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:15:09.660728 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06bc5b4-e294-472f-9e53-704b6b9072f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:15:09.660991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06bc5b4-e294-472f-9e53-704b6b9072f8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:15:09.661624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06bc5b4-e294-472f-9e53-704b6b9072f8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:15:09.661916 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06bc5b4-e294-472f-9e53-704b6b9072f8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:15:09.662937 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: WARNING py.warnings [None req-b06bc5b4-e294-472f-9e53-704b6b9072f8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 13 01:15:09.662937 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: warnings.warn(msg) Sep 13 01:15:09.662937 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: Sep 13 01:15:09.663810 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06bc5b4-e294-472f-9e53-704b6b9072f8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:15:09.674558 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 743/1496] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:15:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:15:09.682876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9917948d-0574-4d71-9b50-a4dcf3aed1b7 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:15:09.692573 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9917948d-0574-4d71-9b50-a4dcf3aed1b7 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:15:09.753852 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:15:09.786319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336f5422-8a5a-43ef-b26e-9c1196fbeea3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:15:09.786946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336f5422-8a5a-43ef-b26e-9c1196fbeea3 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:15:09.787224 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336f5422-8a5a-43ef-b26e-9c1196fbeea3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:15:09.787491 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336f5422-8a5a-43ef-b26e-9c1196fbeea3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:15:09.788100 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336f5422-8a5a-43ef-b26e-9c1196fbeea3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:15:09.789829 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: WARNING py.warnings [None req-336f5422-8a5a-43ef-b26e-9c1196fbeea3 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 01:15:09.789829 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: warnings.warn(msg) Sep 13 01:15:09.789829 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: Sep 13 01:15:09.791106 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336f5422-8a5a-43ef-b26e-9c1196fbeea3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:15:09.848018 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 754/1497] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:15:09 2022] POST /identity/v3/projects => generated 379 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:15:09.852301 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06bc5b4-e294-472f-9e53-704b6b9072f8 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:15:09.854136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b06bc5b4-e294-472f-9e53-704b6b9072f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:15:09.887449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06bc5b4-e294-472f-9e53-704b6b9072f8 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:15:09.989903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f24fa5-07da-48b1-9d08-ffcc3a4e5055 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:15:09.990513 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f24fa5-07da-48b1-9d08-ffcc3a4e5055 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:15:09.990791 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f24fa5-07da-48b1-9d08-ffcc3a4e5055 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:15:09.991091 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f24fa5-07da-48b1-9d08-ffcc3a4e5055 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:15:09.991723 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f24fa5-07da-48b1-9d08-ffcc3a4e5055 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:15:09.992727 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: WARNING py.warnings [None req-d5f24fa5-07da-48b1-9d08-ffcc3a4e5055 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 01:15:09.992727 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: warnings.warn(msg) Sep 13 01:15:09.992727 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: Sep 13 01:15:09.993930 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f24fa5-07da-48b1-9d08-ffcc3a4e5055 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:15:10.036625 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 744/1498] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:15:09 2022] POST /identity/v3/users => generated 290 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:15:10.040709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336f5422-8a5a-43ef-b26e-9c1196fbeea3 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:15:10.042259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-336f5422-8a5a-43ef-b26e-9c1196fbeea3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:15:10.105687 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336f5422-8a5a-43ef-b26e-9c1196fbeea3 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:15:10.187628 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:15:10.199408 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc1b0fd-20be-4cff-a60a-cacea75ccf78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:15:10.200014 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc1b0fd-20be-4cff-a60a-cacea75ccf78 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:15:10.200286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc1b0fd-20be-4cff-a60a-cacea75ccf78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:15:10.200552 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc1b0fd-20be-4cff-a60a-cacea75ccf78 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:15:10.201179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc1b0fd-20be-4cff-a60a-cacea75ccf78 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:15:10.202174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: WARNING py.warnings [None req-acc1b0fd-20be-4cff-a60a-cacea75ccf78 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 13 01:15:10.202174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: warnings.warn(msg) Sep 13 01:15:10.202174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: Sep 13 01:15:10.203072 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc1b0fd-20be-4cff-a60a-cacea75ccf78 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:15:10.215317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 755/1499] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:15:10 2022] GET /identity/v3/roles => generated 1420 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:15:10.219309 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f24fa5-07da-48b1-9d08-ffcc3a4e5055 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:15:10.221447 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d5f24fa5-07da-48b1-9d08-ffcc3a4e5055 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:15:10.242010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f24fa5-07da-48b1-9d08-ffcc3a4e5055 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:15:10.271593 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:15:10.341360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b38e5a8-81bd-4d00-b921-0fac129f1e88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:15:10.341941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b38e5a8-81bd-4d00-b921-0fac129f1e88 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:15:10.342216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b38e5a8-81bd-4d00-b921-0fac129f1e88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:15:10.342480 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b38e5a8-81bd-4d00-b921-0fac129f1e88 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:15:10.343142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b38e5a8-81bd-4d00-b921-0fac129f1e88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:15:10.344139 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: WARNING py.warnings [None req-2b38e5a8-81bd-4d00-b921-0fac129f1e88 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 13 01:15:10.344139 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: warnings.warn(msg) Sep 13 01:15:10.344139 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: Sep 13 01:15:10.345522 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b38e5a8-81bd-4d00-b921-0fac129f1e88 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:15:10.357625 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 745/1500] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:15:10 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:15:10.362506 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc1b0fd-20be-4cff-a60a-cacea75ccf78 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:15:10.364079 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-acc1b0fd-20be-4cff-a60a-cacea75ccf78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:15:10.396801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc1b0fd-20be-4cff-a60a-cacea75ccf78 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:15:10.490354 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9e63f2-4883-48c3-b423-79534870b961 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:15:10.491038 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9e63f2-4883-48c3-b423-79534870b961 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:15:10.491317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9e63f2-4883-48c3-b423-79534870b961 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:15:10.491591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9e63f2-4883-48c3-b423-79534870b961 admin admin] PATH_INFO: `/v3/projects/cea4f974038d4952a36b07fe30d5e6f2/users/9b3ac9eefcdd4e27a38bd1d2c3db9d25/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:15:10.513415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9e63f2-4883-48c3-b423-79534870b961 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cea4f974038d4952a36b07fe30d5e6f2, user_id=9b3ac9eefcdd4e27a38bd1d2c3db9d25, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:15:10.514428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: WARNING py.warnings [None req-ce9e63f2-4883-48c3-b423-79534870b961 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 01:15:10.514428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: warnings.warn(msg) Sep 13 01:15:10.514428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: Sep 13 01:15:10.516409 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9e63f2-4883-48c3-b423-79534870b961 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:15:10.541282 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 756/1501] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:15:10 2022] PUT /identity/v3/projects/cea4f974038d4952a36b07fe30d5e6f2/users/9b3ac9eefcdd4e27a38bd1d2c3db9d25/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:15:10.553906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c44c9c-facd-4b2e-ae0c-f05cd830541a None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:15:10.554195 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c44c9c-facd-4b2e-ae0c-f05cd830541a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:15:10.554474 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c44c9c-facd-4b2e-ae0c-f05cd830541a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:15:10.609967 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-40c44c9c-facd-4b2e-ae0c-f05cd830541a None None] MFA Rules not processed for user `9b3ac9eefcdd4e27a38bd1d2c3db9d25`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:15:10.611636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-40c44c9c-facd-4b2e-ae0c-f05cd830541a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:15:10.686595 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 746/1502] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:15:10 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:15:10.700363 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6cb916-8f64-4482-acd5-ae968c3ce547 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:15:10.700654 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6cb916-8f64-4482-acd5-ae968c3ce547 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:15:10.700933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6cb916-8f64-4482-acd5-ae968c3ce547 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:15:10.702265 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 757/1503] 10.222.0.15 () {54 vars in 964 bytes} [Tue Sep 13 01:15:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:15:10.707086 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38f1627-290c-4edc-89ef-f4b0e496dcac None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:15:10.707372 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38f1627-290c-4edc-89ef-f4b0e496dcac None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:15:10.707671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38f1627-290c-4edc-89ef-f4b0e496dcac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:15:10.739069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:15:10.784018 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c38f1627-290c-4edc-89ef-f4b0e496dcac None None] MFA Rules not processed for user `f90a5e57daa643eab94c01e21fcbc4c4`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:15:10.785223 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c38f1627-290c-4edc-89ef-f4b0e496dcac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:15:10.859405 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 747/1504] 10.222.0.15 () {58 vars in 1061 bytes} [Tue Sep 13 01:15:10 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:15:10.867631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6cb916-8f64-4482-acd5-ae968c3ce547 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:15:10.869080 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0d6cb916-8f64-4482-acd5-ae968c3ce547 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:15:10.919924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:15:10.976253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6cb916-8f64-4482-acd5-ae968c3ce547 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:15:11.049877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf9cf37-a1cc-4550-a011-2f5e6848f56d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:15:11.050479 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf9cf37-a1cc-4550-a011-2f5e6848f56d service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:15:11.050756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf9cf37-a1cc-4550-a011-2f5e6848f56d service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:15:11.051057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf9cf37-a1cc-4550-a011-2f5e6848f56d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:15:11.052664 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-caf9cf37-a1cc-4550-a011-2f5e6848f56d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:15:11.148185 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf9cf37-a1cc-4550-a011-2f5e6848f56d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:15:11.149523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf9cf37-a1cc-4550-a011-2f5e6848f56d service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:15:11.152932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:15:11.167812 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 758/1505] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:15:10 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:15:11.213020 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c603d17d-cd0d-4df2-8ab7-2b2bee729a32 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:15:11.213311 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c603d17d-cd0d-4df2-8ab7-2b2bee729a32 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:15:11.213591 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c603d17d-cd0d-4df2-8ab7-2b2bee729a32 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:15:11.214865 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 748/1506] 10.222.0.15 () {52 vars in 830 bytes} [Tue Sep 13 01:15:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:15:11.218445 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf9cf37-a1cc-4550-a011-2f5e6848f56d service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:15:11.220215 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-caf9cf37-a1cc-4550-a011-2f5e6848f56d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:15:11.318443 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf9cf37-a1cc-4550-a011-2f5e6848f56d service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:15:11.411039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d1f25c-077b-4ea9-a2ee-07b4582fca06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:15:11.411676 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d1f25c-077b-4ea9-a2ee-07b4582fca06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:15:11.411954 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d1f25c-077b-4ea9-a2ee-07b4582fca06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:15:11.412229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d1f25c-077b-4ea9-a2ee-07b4582fca06 admin admin] PATH_INFO: `/v3/users/9b3ac9eefcdd4e27a38bd1d2c3db9d25` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:15:11.425325 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d1f25c-077b-4ea9-a2ee-07b4582fca06 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9b3ac9eefcdd4e27a38bd1d2c3db9d25)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:15:11.426350 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: WARNING py.warnings [None req-b2d1f25c-077b-4ea9-a2ee-07b4582fca06 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 01:15:11.426350 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: warnings.warn(msg) Sep 13 01:15:11.426350 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: Sep 13 01:15:11.427593 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d1f25c-077b-4ea9-a2ee-07b4582fca06 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:15:11.437448 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:15:11.495064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b2d1f25c-077b-4ea9-a2ee-07b4582fca06 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9b3ac9eefcdd4e27a38bd1d2c3db9d25'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:15:11.495594 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b2d1f25c-077b-4ea9-a2ee-07b4582fca06 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9b3ac9eefcdd4e27a38bd1d2c3db9d25'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:15:11.504456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b2d1f25c-077b-4ea9-a2ee-07b4582fca06 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9b3ac9eefcdd4e27a38bd1d2c3db9d25'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:15:11.504914 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b2d1f25c-077b-4ea9-a2ee-07b4582fca06 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9b3ac9eefcdd4e27a38bd1d2c3db9d25'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:15:11.529570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b2d1f25c-077b-4ea9-a2ee-07b4582fca06 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9b3ac9eefcdd4e27a38bd1d2c3db9d25'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:15:11.548678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 759/1507] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:15:11 2022] DELETE /identity/v3/users/9b3ac9eefcdd4e27a38bd1d2c3db9d25 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:15:11.553012 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6ebf50-b5c7-4344-aad6-5ddc1f45cf17 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:15:11.553483 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6ebf50-b5c7-4344-aad6-5ddc1f45cf17 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:15:11.553587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6ebf50-b5c7-4344-aad6-5ddc1f45cf17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:15:11.615425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8f6ebf50-b5c7-4344-aad6-5ddc1f45cf17 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:15:11.616665 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8f6ebf50-b5c7-4344-aad6-5ddc1f45cf17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:15:11.697121 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:15:11.710127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 749/1508] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:15:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:15:11.719730 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d1f25c-077b-4ea9-a2ee-07b4582fca06 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:15:11.720950 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b2d1f25c-077b-4ea9-a2ee-07b4582fca06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:15:11.819002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d1f25c-077b-4ea9-a2ee-07b4582fca06 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:15:11.872433 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:15:11.892975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae26b790-65cc-49bc-af0f-25bd1af0cb8d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:15:11.893584 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae26b790-65cc-49bc-af0f-25bd1af0cb8d service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:15:11.893863 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae26b790-65cc-49bc-af0f-25bd1af0cb8d service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:15:11.894134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae26b790-65cc-49bc-af0f-25bd1af0cb8d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:15:11.895740 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ae26b790-65cc-49bc-af0f-25bd1af0cb8d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:15:12.003918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae26b790-65cc-49bc-af0f-25bd1af0cb8d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:15:12.005344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae26b790-65cc-49bc-af0f-25bd1af0cb8d service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:15:12.023457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 760/1509] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:15:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:15:12.452096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6ebf50-b5c7-4344-aad6-5ddc1f45cf17 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:15:12.453419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8f6ebf50-b5c7-4344-aad6-5ddc1f45cf17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:15:12.468606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6ebf50-b5c7-4344-aad6-5ddc1f45cf17 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:15:12.568567 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd757dd-4174-4490-bdca-890ae78915ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:15:12.569197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd757dd-4174-4490-bdca-890ae78915ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:15:12.569474 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd757dd-4174-4490-bdca-890ae78915ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:15:12.569745 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd757dd-4174-4490-bdca-890ae78915ae admin admin] PATH_INFO: `/v3/projects/cea4f974038d4952a36b07fe30d5e6f2` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:15:12.579615 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd757dd-4174-4490-bdca-890ae78915ae admin admin] RBAC: Authorizing `identity:delete_project(project_id=cea4f974038d4952a36b07fe30d5e6f2)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:15:12.580633 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: WARNING py.warnings [None req-1bd757dd-4174-4490-bdca-890ae78915ae admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 01:15:12.580633 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: warnings.warn(msg) Sep 13 01:15:12.580633 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: Sep 13 01:15:12.581823 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd757dd-4174-4490-bdca-890ae78915ae admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:15:12.638199 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1bd757dd-4174-4490-bdca-890ae78915ae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cea4f974038d4952a36b07fe30d5e6f2', 'request_id': 'req-1bd757dd-4174-4490-bdca-890ae78915ae'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:15:12.644255 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1bd757dd-4174-4490-bdca-890ae78915ae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cea4f974038d4952a36b07fe30d5e6f2', 'request_id': 'req-1bd757dd-4174-4490-bdca-890ae78915ae'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:15:12.658802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1bd757dd-4174-4490-bdca-890ae78915ae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cea4f974038d4952a36b07fe30d5e6f2', 'request_id': 'req-1bd757dd-4174-4490-bdca-890ae78915ae'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:15:12.684199 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1bd757dd-4174-4490-bdca-890ae78915ae admin admin] The token cache is being invalidate because project cea4f974038d4952a36b07fe30d5e6f2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:15:12.684489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1bd757dd-4174-4490-bdca-890ae78915ae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:15:12.685259 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1bd757dd-4174-4490-bdca-890ae78915ae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:15:12.686544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 750/1510] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:15:12 2022] DELETE /identity/v3/projects/cea4f974038d4952a36b07fe30d5e6f2 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:15:22.021207 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=62 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:15:22.682243 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:16:37.473615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2493075-1e98-47bc-a6a8-e3aae09b319b None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:37.474208 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2493075-1e98-47bc-a6a8-e3aae09b319b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:37.474208 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2493075-1e98-47bc-a6a8-e3aae09b319b None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:37.475522 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 761/1511] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 01:16:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:16:37.480269 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7629d751-84b8-49b7-ac00-62b710a63e2d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:37.480671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7629d751-84b8-49b7-ac00-62b710a63e2d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:37.480815 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7629d751-84b8-49b7-ac00-62b710a63e2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:37.564096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7629d751-84b8-49b7-ac00-62b710a63e2d None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:16:37.565299 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7629d751-84b8-49b7-ac00-62b710a63e2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:16:37.667911 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 751/1512] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 01:16:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:16:37.824869 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2493075-1e98-47bc-a6a8-e3aae09b319b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:16:37.826953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a2493075-1e98-47bc-a6a8-e3aae09b319b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:16:37.944874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2493075-1e98-47bc-a6a8-e3aae09b319b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:16:37.950465 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae25d0cc-d28f-459c-af2d-94cf2cc7dbbb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:16:37.951090 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae25d0cc-d28f-459c-af2d-94cf2cc7dbbb service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:37.951364 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae25d0cc-d28f-459c-af2d-94cf2cc7dbbb service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:37.951632 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae25d0cc-d28f-459c-af2d-94cf2cc7dbbb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:37.960436 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae25d0cc-d28f-459c-af2d-94cf2cc7dbbb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:16:37.961998 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae25d0cc-d28f-459c-af2d-94cf2cc7dbbb service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:16:37.971519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 762/1513] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:16:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:16:38.974037 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0c95a4-762c-4443-b4b6-d3aa26df2c89 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:38.974603 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0c95a4-762c-4443-b4b6-d3aa26df2c89 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:38.974603 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0c95a4-762c-4443-b4b6-d3aa26df2c89 None None] PATH_INFO: `/v3` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:38.975988 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 752/1514] 10.222.0.25 () {54 vars in 925 bytes} [Tue Sep 13 01:16:38 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:16:38.982199 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a5f033-5da4-4d8a-be22-1e5f472ea927 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:38.982700 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a5f033-5da4-4d8a-be22-1e5f472ea927 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:38.982885 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a5f033-5da4-4d8a-be22-1e5f472ea927 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:39.042319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e7a5f033-5da4-4d8a-be22-1e5f472ea927 None None] MFA Rules not processed for user `8eaff1b843a145a49994dcf089f44315`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:16:39.043528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e7a5f033-5da4-4d8a-be22-1e5f472ea927 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:16:39.064037 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10001ce-e655-47e7-826b-0704f9098902 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:39.064564 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10001ce-e655-47e7-826b-0704f9098902 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:39.065150 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10001ce-e655-47e7-826b-0704f9098902 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:39.066484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 753/1515] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 01:16:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:16:39.078657 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c994a1e-f393-4a53-a9ae-fbf47be2e2b6 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:39.079057 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c994a1e-f393-4a53-a9ae-fbf47be2e2b6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:39.079399 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c994a1e-f393-4a53-a9ae-fbf47be2e2b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:39.099421 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8c994a1e-f393-4a53-a9ae-fbf47be2e2b6 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:16:39.100618 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8c994a1e-f393-4a53-a9ae-fbf47be2e2b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:16:39.108973 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 754/1516] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 01:16:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:16:39.115907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:16:39.121686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 763/1517] 10.222.0.25 () {58 vars in 1015 bytes} [Tue Sep 13 01:16:38 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:16:39.260682 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c994a1e-f393-4a53-a9ae-fbf47be2e2b6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:16:39.272872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c994a1e-f393-4a53-a9ae-fbf47be2e2b6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:16:39.278659 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00a3e24-83c0-46bc-837f-e5c0ef1158c5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:16:39.279298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00a3e24-83c0-46bc-837f-e5c0ef1158c5 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:39.279579 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00a3e24-83c0-46bc-837f-e5c0ef1158c5 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:39.279851 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00a3e24-83c0-46bc-837f-e5c0ef1158c5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:39.288061 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00a3e24-83c0-46bc-837f-e5c0ef1158c5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:16:39.289634 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00a3e24-83c0-46bc-837f-e5c0ef1158c5 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:16:39.299441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 755/1518] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:16:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:16:42.634245 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d310a1a-132c-42ad-b7c2-068197d960fa None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:42.634681 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d310a1a-132c-42ad-b7c2-068197d960fa None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:42.634792 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d310a1a-132c-42ad-b7c2-068197d960fa None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:42.636196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 764/1519] 10.222.0.15 () {54 vars in 961 bytes} [Tue Sep 13 01:16:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:16:42.642124 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c5cd8b-8bef-4870-873c-e28cd4977b15 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:42.642573 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c5cd8b-8bef-4870-873c-e28cd4977b15 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:42.642746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c5cd8b-8bef-4870-873c-e28cd4977b15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:42.662081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-22c5cd8b-8bef-4870-873c-e28cd4977b15 None None] MFA Rules not processed for user `8eaff1b843a145a49994dcf089f44315`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:16:42.664032 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-22c5cd8b-8bef-4870-873c-e28cd4977b15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:16:42.672626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 756/1520] 10.222.0.15 () {58 vars in 1058 bytes} [Tue Sep 13 01:16:42 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:16:42.681428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d310a1a-132c-42ad-b7c2-068197d960fa None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:16:42.696667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d310a1a-132c-42ad-b7c2-068197d960fa None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:16:42.701580 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f62351-3047-4a84-a2a3-051e3f82cf3d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8eaff1b843a145a49994dcf089f44315', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:16:42.702757 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f62351-3047-4a84-a2a3-051e3f82cf3d service placement] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:42.703064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f62351-3047-4a84-a2a3-051e3f82cf3d service placement] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:42.703337 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f62351-3047-4a84-a2a3-051e3f82cf3d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:42.711493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f62351-3047-4a84-a2a3-051e3f82cf3d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:16:42.713055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f62351-3047-4a84-a2a3-051e3f82cf3d service placement] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:16:42.722580 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 765/1521] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:16:42 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:16:50.765522 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32a31a9-7e96-44f9-bf08-eb444fb43302 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:50.766074 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32a31a9-7e96-44f9-bf08-eb444fb43302 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:50.766074 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32a31a9-7e96-44f9-bf08-eb444fb43302 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:50.767488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 757/1522] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 01:16:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:16:50.773446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907be8e3-559b-4dc0-9336-3e7e14fbd905 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:50.773847 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907be8e3-559b-4dc0-9336-3e7e14fbd905 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:50.773995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907be8e3-559b-4dc0-9336-3e7e14fbd905 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:50.793634 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-907be8e3-559b-4dc0-9336-3e7e14fbd905 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:16:50.794815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-907be8e3-559b-4dc0-9336-3e7e14fbd905 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:16:50.803229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 766/1523] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 01:16:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:16:50.956588 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32a31a9-7e96-44f9-bf08-eb444fb43302 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:16:50.971656 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32a31a9-7e96-44f9-bf08-eb444fb43302 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:16:50.977003 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fb8358-dcbe-4bd4-8fc4-a2e847e8d01b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:16:50.977596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fb8358-dcbe-4bd4-8fc4-a2e847e8d01b service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:50.977875 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fb8358-dcbe-4bd4-8fc4-a2e847e8d01b service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:50.978145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fb8358-dcbe-4bd4-8fc4-a2e847e8d01b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:50.986744 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39fb8358-dcbe-4bd4-8fc4-a2e847e8d01b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:16:50.988340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39fb8358-dcbe-4bd4-8fc4-a2e847e8d01b service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:16:50.997579 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 758/1524] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:16:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:16:52.071023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d24114-0266-4737-ac6d-c5d8c5fb14d4 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:52.071596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d24114-0266-4737-ac6d-c5d8c5fb14d4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:52.071596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d24114-0266-4737-ac6d-c5d8c5fb14d4 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:52.072899 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 767/1525] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 01:16:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:16:52.079481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2717ee06-92b0-4657-a19f-e74d76865a5b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:52.079923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2717ee06-92b0-4657-a19f-e74d76865a5b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:52.080027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2717ee06-92b0-4657-a19f-e74d76865a5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:52.099620 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-2717ee06-92b0-4657-a19f-e74d76865a5b None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:16:52.100814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2717ee06-92b0-4657-a19f-e74d76865a5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:16:52.108602 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 759/1526] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 01:16:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:16:52.260224 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d24114-0266-4737-ac6d-c5d8c5fb14d4 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:16:52.274320 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d24114-0266-4737-ac6d-c5d8c5fb14d4 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:16:52.279672 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4902405c-2483-4de3-bd10-e3697dcae318 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:16:52.280780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4902405c-2483-4de3-bd10-e3697dcae318 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:52.281057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4902405c-2483-4de3-bd10-e3697dcae318 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:52.281325 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4902405c-2483-4de3-bd10-e3697dcae318 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:52.290667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4902405c-2483-4de3-bd10-e3697dcae318 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:16:52.292260 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4902405c-2483-4de3-bd10-e3697dcae318 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:16:52.302002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 768/1527] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:16:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:16:56.298143 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2180888-911e-42d1-8161-1532614771d7 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:56.298716 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2180888-911e-42d1-8161-1532614771d7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:56.298716 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2180888-911e-42d1-8161-1532614771d7 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:56.300068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 760/1528] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 01:16:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:16:56.305182 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b178775-cc03-4cfa-9a31-c08b23f2c3e8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:56.305575 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b178775-cc03-4cfa-9a31-c08b23f2c3e8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:56.305734 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b178775-cc03-4cfa-9a31-c08b23f2c3e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:56.326922 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-6b178775-cc03-4cfa-9a31-c08b23f2c3e8 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:16:56.328132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6b178775-cc03-4cfa-9a31-c08b23f2c3e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:16:56.336019 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 769/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 01:16:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:16:56.623130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2180888-911e-42d1-8161-1532614771d7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:16:56.625305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f2180888-911e-42d1-8161-1532614771d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:16:56.652349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:16:56.704571 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2180888-911e-42d1-8161-1532614771d7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:16:56.708120 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699aa149-9b80-4b67-81a2-9bf5724f9892 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:16:56.708717 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699aa149-9b80-4b67-81a2-9bf5724f9892 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:56.708996 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699aa149-9b80-4b67-81a2-9bf5724f9892 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:56.709264 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699aa149-9b80-4b67-81a2-9bf5724f9892 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:56.718274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699aa149-9b80-4b67-81a2-9bf5724f9892 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:16:56.719726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699aa149-9b80-4b67-81a2-9bf5724f9892 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:16:56.728926 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 761/1530] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:16:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:16:58.865919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b455fdd3-ac36-451e-9b08-32241681463b None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:58.866463 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b455fdd3-ac36-451e-9b08-32241681463b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:58.866463 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b455fdd3-ac36-451e-9b08-32241681463b None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:58.867864 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 770/1531] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 01:16:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:16:58.872952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5a3235-1c06-4d4f-9c39-d575858493de None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:58.873425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5a3235-1c06-4d4f-9c39-d575858493de None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:58.874098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5a3235-1c06-4d4f-9c39-d575858493de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:58.893544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-1a5a3235-1c06-4d4f-9c39-d575858493de None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:16:58.894726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1a5a3235-1c06-4d4f-9c39-d575858493de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:16:58.902362 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 762/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 01:16:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:16:59.186730 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b455fdd3-ac36-451e-9b08-32241681463b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:16:59.197304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b455fdd3-ac36-451e-9b08-32241681463b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:16:59.200757 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99e8743-9ae5-4311-ace2-b38647b2fec7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:16:59.201342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99e8743-9ae5-4311-ace2-b38647b2fec7 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:16:59.201622 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99e8743-9ae5-4311-ace2-b38647b2fec7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:16:59.201898 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99e8743-9ae5-4311-ace2-b38647b2fec7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:16:59.210526 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99e8743-9ae5-4311-ace2-b38647b2fec7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:16:59.211985 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99e8743-9ae5-4311-ace2-b38647b2fec7 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:16:59.221659 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 771/1533] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:16:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:17:02.917817 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75865ab8-8d0d-4d2c-8725-9704b95a0000 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:17:02.918340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75865ab8-8d0d-4d2c-8725-9704b95a0000 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:17:02.918340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75865ab8-8d0d-4d2c-8725-9704b95a0000 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:17:02.919683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 763/1534] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 01:17:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:17:02.925515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc29a61-6722-4a43-9fe3-e31dcd23deac None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:17:02.925954 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc29a61-6722-4a43-9fe3-e31dcd23deac None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:17:02.926051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc29a61-6722-4a43-9fe3-e31dcd23deac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:17:02.945211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-bfc29a61-6722-4a43-9fe3-e31dcd23deac None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:17:02.946385 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-bfc29a61-6722-4a43-9fe3-e31dcd23deac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:17:02.954819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 772/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 01:17:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:17:03.256868 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75865ab8-8d0d-4d2c-8725-9704b95a0000 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:17:03.258524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-75865ab8-8d0d-4d2c-8725-9704b95a0000 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:17:03.344547 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75865ab8-8d0d-4d2c-8725-9704b95a0000 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:17:03.348148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1498fc-9ac1-4ca1-94dc-0ad4da57b3ca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:17:03.348749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1498fc-9ac1-4ca1-94dc-0ad4da57b3ca service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:17:03.349051 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1498fc-9ac1-4ca1-94dc-0ad4da57b3ca service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:17:03.349319 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1498fc-9ac1-4ca1-94dc-0ad4da57b3ca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:17:03.358472 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1498fc-9ac1-4ca1-94dc-0ad4da57b3ca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:17:03.359990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1498fc-9ac1-4ca1-94dc-0ad4da57b3ca service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:17:03.369366 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 764/1536] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:17:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:17:05.489533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2409054b-7a76-4bfb-bd54-4bbde6ad350e None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:17:05.490078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2409054b-7a76-4bfb-bd54-4bbde6ad350e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:17:05.490078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2409054b-7a76-4bfb-bd54-4bbde6ad350e None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:17:05.491487 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 773/1537] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 01:17:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:17:05.496810 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac32818-48d6-4a6f-87be-12cb0acd5fca None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:17:05.497247 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac32818-48d6-4a6f-87be-12cb0acd5fca None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:17:05.497369 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac32818-48d6-4a6f-87be-12cb0acd5fca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:17:05.517106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7ac32818-48d6-4a6f-87be-12cb0acd5fca None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:17:05.518283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7ac32818-48d6-4a6f-87be-12cb0acd5fca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:17:05.525934 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 765/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 01:17:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:17:05.832098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2409054b-7a76-4bfb-bd54-4bbde6ad350e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:17:05.833777 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2409054b-7a76-4bfb-bd54-4bbde6ad350e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:17:05.846513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2409054b-7a76-4bfb-bd54-4bbde6ad350e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:17:05.850060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b77b9e-99f2-4dbf-8e1c-d92ebfd82894 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:17:05.850672 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b77b9e-99f2-4dbf-8e1c-d92ebfd82894 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:17:05.850985 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b77b9e-99f2-4dbf-8e1c-d92ebfd82894 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:17:05.851261 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b77b9e-99f2-4dbf-8e1c-d92ebfd82894 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:17:05.860039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b77b9e-99f2-4dbf-8e1c-d92ebfd82894 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:17:05.861441 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b77b9e-99f2-4dbf-8e1c-d92ebfd82894 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:17:05.870771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 774/1539] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:17:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:17:15.512263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=66 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:17:15.866628 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=34 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:29.138445 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611b90b0-9e38-425f-867c-6796921ef331 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.139129 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611b90b0-9e38-425f-867c-6796921ef331 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.139129 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611b90b0-9e38-425f-867c-6796921ef331 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.165419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-611b90b0-9e38-425f-867c-6796921ef331 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:29.166675 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-611b90b0-9e38-425f-867c-6796921ef331 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:29.181471 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 766/1540] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:29.191859 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e256b9b3-5b19-4386-b62d-b7d8be9fd12f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.193500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e256b9b3-5b19-4386-b62d-b7d8be9fd12f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.193500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e256b9b3-5b19-4386-b62d-b7d8be9fd12f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.220731 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e256b9b3-5b19-4386-b62d-b7d8be9fd12f None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:29.222025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e256b9b3-5b19-4386-b62d-b7d8be9fd12f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:29.238039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 775/1541] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:29.243033 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611b90b0-9e38-425f-867c-6796921ef331 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:29.254414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611b90b0-9e38-425f-867c-6796921ef331 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:29.264438 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda14982-f3d6-4106-b053-670503fa3145 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:29.265073 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda14982-f3d6-4106-b053-670503fa3145 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.265348 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda14982-f3d6-4106-b053-670503fa3145 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.265616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda14982-f3d6-4106-b053-670503fa3145 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.267087 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda14982-f3d6-4106-b053-670503fa3145 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:18:29.267387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda14982-f3d6-4106-b053-670503fa3145 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:29.269007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda14982-f3d6-4106-b053-670503fa3145 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:29.288548 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 767/1542] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:18:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:18:29.296495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e256b9b3-5b19-4386-b62d-b7d8be9fd12f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:29.311873 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e256b9b3-5b19-4386-b62d-b7d8be9fd12f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:29.319265 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd4b984-ad22-4d39-a6ec-6869f1ab769d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:29.320252 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd4b984-ad22-4d39-a6ec-6869f1ab769d admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.320530 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd4b984-ad22-4d39-a6ec-6869f1ab769d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.320797 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd4b984-ad22-4d39-a6ec-6869f1ab769d admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.321444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd4b984-ad22-4d39-a6ec-6869f1ab769d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:29.323356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd4b984-ad22-4d39-a6ec-6869f1ab769d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:29.380525 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 776/1543] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:29 2022] POST /identity/v3/projects => generated 377 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:29.384831 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda14982-f3d6-4106-b053-670503fa3145 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:29.395716 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda14982-f3d6-4106-b053-670503fa3145 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:29.417033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295619ad-9c2e-4cde-a5c1-bf675f5a9e4c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.417316 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295619ad-9c2e-4cde-a5c1-bf675f5a9e4c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.417589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295619ad-9c2e-4cde-a5c1-bf675f5a9e4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.448317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-295619ad-9c2e-4cde-a5c1-bf675f5a9e4c None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:29.449518 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-295619ad-9c2e-4cde-a5c1-bf675f5a9e4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:29.486241 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54cfba1-f8e4-4cba-9d88-b65371b9caf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:29.487294 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54cfba1-f8e4-4cba-9d88-b65371b9caf5 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.487796 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54cfba1-f8e4-4cba-9d88-b65371b9caf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.488290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54cfba1-f8e4-4cba-9d88-b65371b9caf5 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.489147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54cfba1-f8e4-4cba-9d88-b65371b9caf5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:29.491413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54cfba1-f8e4-4cba-9d88-b65371b9caf5 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:29.532540 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 768/1544] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:29 2022] POST /identity/v3/users => generated 289 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:29.536801 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54cfba1-f8e4-4cba-9d88-b65371b9caf5 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:29.556048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54cfba1-f8e4-4cba-9d88-b65371b9caf5 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:29.559736 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 777/1545] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Sep 13 01:18:29.561480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d5b694-6110-4188-93e8-0405bdfccf31 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.562286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d5b694-6110-4188-93e8-0405bdfccf31 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.562760 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b313f086-f56e-4299-91e7-f9c0e45d9d22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:29.563440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d5b694-6110-4188-93e8-0405bdfccf31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.564095 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b313f086-f56e-4299-91e7-f9c0e45d9d22 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.564527 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b313f086-f56e-4299-91e7-f9c0e45d9d22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.564892 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b313f086-f56e-4299-91e7-f9c0e45d9d22 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.565645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b313f086-f56e-4299-91e7-f9c0e45d9d22 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:29.567587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b313f086-f56e-4299-91e7-f9c0e45d9d22 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:29.588114 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 769/1546] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:29 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:18:29.590331 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd76742c-413f-4d68-93c9-d2bca97345de None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.590762 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd76742c-413f-4d68-93c9-d2bca97345de None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.591178 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd76742c-413f-4d68-93c9-d2bca97345de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.601165 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-58d5b694-6110-4188-93e8-0405bdfccf31 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:29.603027 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-58d5b694-6110-4188-93e8-0405bdfccf31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:29.612743 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-bd76742c-413f-4d68-93c9-d2bca97345de None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:29.614194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-bd76742c-413f-4d68-93c9-d2bca97345de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:29.616831 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 778/1547] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:29.618103 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f010374-9185-44d5-97c4-f59f96ff5cc8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.618346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f010374-9185-44d5-97c4-f59f96ff5cc8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.618614 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f010374-9185-44d5-97c4-f59f96ff5cc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.629032 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 770/1548] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:29.629862 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd76742c-413f-4d68-93c9-d2bca97345de None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:29.644616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4f010374-9185-44d5-97c4-f59f96ff5cc8 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:29.645881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4f010374-9185-44d5-97c4-f59f96ff5cc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:29.647988 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd76742c-413f-4d68-93c9-d2bca97345de None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:29.653040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d138ed7-da60-4ef1-b6d2-6107aa01c972 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:29.653663 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d138ed7-da60-4ef1-b6d2-6107aa01c972 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.654311 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d138ed7-da60-4ef1-b6d2-6107aa01c972 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.654548 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 779/1549] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:29.654646 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d138ed7-da60-4ef1-b6d2-6107aa01c972 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.655330 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d138ed7-da60-4ef1-b6d2-6107aa01c972 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:29.655719 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07aa8455-cbd9-4f1f-bb81-e7a4bc615977 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.655999 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07aa8455-cbd9-4f1f-bb81-e7a4bc615977 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.656269 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07aa8455-cbd9-4f1f-bb81-e7a4bc615977 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.657206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d138ed7-da60-4ef1-b6d2-6107aa01c972 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:29.668816 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 771/1550] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:29 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:29.670063 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b677fe6-38c5-4a9b-a19e-4aa88ce918c3 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.670348 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b677fe6-38c5-4a9b-a19e-4aa88ce918c3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.670621 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b677fe6-38c5-4a9b-a19e-4aa88ce918c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.683358 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-07aa8455-cbd9-4f1f-bb81-e7a4bc615977 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:29.684593 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-07aa8455-cbd9-4f1f-bb81-e7a4bc615977 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:29.691424 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8b677fe6-38c5-4a9b-a19e-4aa88ce918c3 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:29.692583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8b677fe6-38c5-4a9b-a19e-4aa88ce918c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:29.693352 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 780/1551] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:29.694423 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e4658d-e708-43a8-9d63-f691549ae52e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.694702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e4658d-e708-43a8-9d63-f691549ae52e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.695007 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e4658d-e708-43a8-9d63-f691549ae52e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.706205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 772/1552] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:29.707687 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e08333-c30b-4d86-9462-a5ddd97e7873 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.707973 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e08333-c30b-4d86-9462-a5ddd97e7873 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.708243 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e08333-c30b-4d86-9462-a5ddd97e7873 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.716293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-40e4658d-e708-43a8-9d63-f691549ae52e None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:29.717488 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-40e4658d-e708-43a8-9d63-f691549ae52e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:29.725447 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 781/1553] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:29.726220 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e4658d-e708-43a8-9d63-f691549ae52e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:29.728229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-01e08333-c30b-4d86-9462-a5ddd97e7873 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:29.729482 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-01e08333-c30b-4d86-9462-a5ddd97e7873 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:29.735409 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e4658d-e708-43a8-9d63-f691549ae52e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:29.737277 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 773/1554] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:29.738078 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e08333-c30b-4d86-9462-a5ddd97e7873 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:29.739995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8023e12-ccee-4000-a169-943a8435f559 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:29.740581 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8023e12-ccee-4000-a169-943a8435f559 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.740861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8023e12-ccee-4000-a169-943a8435f559 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.741133 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8023e12-ccee-4000-a169-943a8435f559 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.741763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8023e12-ccee-4000-a169-943a8435f559 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:18:29.742064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8023e12-ccee-4000-a169-943a8435f559 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:29.743090 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: WARNING py.warnings [None req-e8023e12-ccee-4000-a169-943a8435f559 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 13 01:18:29.743090 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: warnings.warn(msg) Sep 13 01:18:29.743090 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: Sep 13 01:18:29.743963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8023e12-ccee-4000-a169-943a8435f559 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:29.747404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e08333-c30b-4d86-9462-a5ddd97e7873 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:29.752389 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2faae474-3804-43e5-974f-a901bc51ba11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:29.753019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faae474-3804-43e5-974f-a901bc51ba11 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.753293 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faae474-3804-43e5-974f-a901bc51ba11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.753561 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faae474-3804-43e5-974f-a901bc51ba11 admin admin] PATH_INFO: `/v3/projects/9056ec686513412a87c292387e25e870/users/349e19300f1e48c3ac44fc34611c7f4a/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.756043 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 782/1555] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:18:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:18:29.757190 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a7a2bd-552f-4ab9-9063-e9a52e6fecdd None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.757473 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a7a2bd-552f-4ab9-9063-e9a52e6fecdd None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.757746 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a7a2bd-552f-4ab9-9063-e9a52e6fecdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.775450 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2faae474-3804-43e5-974f-a901bc51ba11 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9056ec686513412a87c292387e25e870, user_id=349e19300f1e48c3ac44fc34611c7f4a, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:29.776590 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: WARNING py.warnings [None req-2faae474-3804-43e5-974f-a901bc51ba11 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 01:18:29.776590 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: warnings.warn(msg) Sep 13 01:18:29.776590 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: Sep 13 01:18:29.777269 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a7a7a2bd-552f-4ab9-9063-e9a52e6fecdd None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:29.778499 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2faae474-3804-43e5-974f-a901bc51ba11 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:29.778621 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a7a7a2bd-552f-4ab9-9063-e9a52e6fecdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:29.787324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 783/1556] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:29.794500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0455aaf-5f28-4cf7-b0f9-49b4b8d8fd9e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.794500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0455aaf-5f28-4cf7-b0f9-49b4b8d8fd9e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.794500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0455aaf-5f28-4cf7-b0f9-49b4b8d8fd9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.798090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 774/1557] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:29 2022] PUT /identity/v3/projects/9056ec686513412a87c292387e25e870/users/349e19300f1e48c3ac44fc34611c7f4a/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:29.799399 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea425364-02a4-409f-8edb-8a86b9cc99e1 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.799690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea425364-02a4-409f-8edb-8a86b9cc99e1 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.799964 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea425364-02a4-409f-8edb-8a86b9cc99e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.808475 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a0455aaf-5f28-4cf7-b0f9-49b4b8d8fd9e None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:29.809655 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a0455aaf-5f28-4cf7-b0f9-49b4b8d8fd9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:29.820526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ea425364-02a4-409f-8edb-8a86b9cc99e1 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:29.821697 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ea425364-02a4-409f-8edb-8a86b9cc99e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:29.872585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 784/1558] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:29.873910 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa0178e-f6be-4fc5-ae6a-21566238281f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.874195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa0178e-f6be-4fc5-ae6a-21566238281f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.874463 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa0178e-f6be-4fc5-ae6a-21566238281f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.889958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 775/1559] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:29.891197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea425364-02a4-409f-8edb-8a86b9cc99e1 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:29.898504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-1fa0178e-f6be-4fc5-ae6a-21566238281f None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:29.899587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:29.899742 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1fa0178e-f6be-4fc5-ae6a-21566238281f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:29.904224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea425364-02a4-409f-8edb-8a86b9cc99e1 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:29.910266 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 785/1560] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:29.911138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa0178e-f6be-4fc5-ae6a-21566238281f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:29.914946 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f12c18a-afbd-4123-8e9c-2d0a3a90311f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:29.915751 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f12c18a-afbd-4123-8e9c-2d0a3a90311f admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.915866 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f12c18a-afbd-4123-8e9c-2d0a3a90311f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.916133 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f12c18a-afbd-4123-8e9c-2d0a3a90311f admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.917359 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f12c18a-afbd-4123-8e9c-2d0a3a90311f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:18:29.917359 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f12c18a-afbd-4123-8e9c-2d0a3a90311f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:29.918699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f12c18a-afbd-4123-8e9c-2d0a3a90311f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:29.921138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa0178e-f6be-4fc5-ae6a-21566238281f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:29.926312 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1d10cb-3ca8-4680-8c74-c89411704c51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:29.927073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1d10cb-3ca8-4680-8c74-c89411704c51 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.927399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1d10cb-3ca8-4680-8c74-c89411704c51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.927792 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1d10cb-3ca8-4680-8c74-c89411704c51 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.928440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1d10cb-3ca8-4680-8c74-c89411704c51 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:18:29.928733 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1d10cb-3ca8-4680-8c74-c89411704c51 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:29.930302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1d10cb-3ca8-4680-8c74-c89411704c51 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:29.931303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 776/1561] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:18:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:18:29.932054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f12c18a-afbd-4123-8e9c-2d0a3a90311f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:29.942346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f12c18a-afbd-4123-8e9c-2d0a3a90311f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:29.944961 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:29.947174 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6ec0f7-4fa6-46a3-a3bd-e425cf04e5f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:29.947834 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6ec0f7-4fa6-46a3-a3bd-e425cf04e5f4 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.948145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6ec0f7-4fa6-46a3-a3bd-e425cf04e5f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.948496 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6ec0f7-4fa6-46a3-a3bd-e425cf04e5f4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.949187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6ec0f7-4fa6-46a3-a3bd-e425cf04e5f4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:29.950182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: WARNING py.warnings [None req-eb6ec0f7-4fa6-46a3-a3bd-e425cf04e5f4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 01:18:29.950182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: warnings.warn(msg) Sep 13 01:18:29.950182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: Sep 13 01:18:29.951405 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6ec0f7-4fa6-46a3-a3bd-e425cf04e5f4 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:29.955200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 786/1562] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:18:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:18:29.956737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e33e025-08f0-4b80-b434-c3c065e29acd None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.957218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e33e025-08f0-4b80-b434-c3c065e29acd None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.957625 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e33e025-08f0-4b80-b434-c3c065e29acd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:29.982697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5e33e025-08f0-4b80-b434-c3c065e29acd None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:29.986951 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 777/1563] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:29 2022] POST /identity/v3/projects => generated 391 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:29.987122 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5e33e025-08f0-4b80-b434-c3c065e29acd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:29.987222 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff78281-c35e-48e7-9ceb-95b059ff8b93 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:29.987222 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff78281-c35e-48e7-9ceb-95b059ff8b93 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:29.987222 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff78281-c35e-48e7-9ceb-95b059ff8b93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.005035 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6ff78281-c35e-48e7-9ceb-95b059ff8b93 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:30.006280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6ff78281-c35e-48e7-9ceb-95b059ff8b93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:30.047844 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 787/1564] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:30.049380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8c177a-84d5-4eb9-9aa6-5064cfddb62b None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.049671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8c177a-84d5-4eb9-9aa6-5064cfddb62b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.049948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8c177a-84d5-4eb9-9aa6-5064cfddb62b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.069338 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ed8c177a-84d5-4eb9-9aa6-5064cfddb62b None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:30.069695 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 778/1565] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:30.070544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff78281-c35e-48e7-9ceb-95b059ff8b93 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.073619 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ed8c177a-84d5-4eb9-9aa6-5064cfddb62b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:30.081239 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff78281-c35e-48e7-9ceb-95b059ff8b93 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.085620 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 788/1566] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:30.085620 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8c177a-84d5-4eb9-9aa6-5064cfddb62b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.085988 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48da67d2-bbd5-41e3-b4fb-e8340f2bd974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.086595 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48da67d2-bbd5-41e3-b4fb-e8340f2bd974 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.086911 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48da67d2-bbd5-41e3-b4fb-e8340f2bd974 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.087187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48da67d2-bbd5-41e3-b4fb-e8340f2bd974 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.087826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48da67d2-bbd5-41e3-b4fb-e8340f2bd974 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:18:30.088128 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48da67d2-bbd5-41e3-b4fb-e8340f2bd974 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.089721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48da67d2-bbd5-41e3-b4fb-e8340f2bd974 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.095311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8c177a-84d5-4eb9-9aa6-5064cfddb62b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.099943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e221e62c-47d0-42aa-9af3-bb1c0d37a0e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.100532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e221e62c-47d0-42aa-9af3-bb1c0d37a0e0 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.100810 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e221e62c-47d0-42aa-9af3-bb1c0d37a0e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.101084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e221e62c-47d0-42aa-9af3-bb1c0d37a0e0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.108898 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 779/1567] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:18:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:18:30.109049 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e221e62c-47d0-42aa-9af3-bb1c0d37a0e0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:18:30.109049 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e221e62c-47d0-42aa-9af3-bb1c0d37a0e0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.109269 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48da67d2-bbd5-41e3-b4fb-e8340f2bd974 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.109377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e221e62c-47d0-42aa-9af3-bb1c0d37a0e0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.118899 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48da67d2-bbd5-41e3-b4fb-e8340f2bd974 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.126738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 789/1568] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:18:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:18:30.126738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e221e62c-47d0-42aa-9af3-bb1c0d37a0e0 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.127133 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8536689-e49e-431a-aafd-280839effc13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.127133 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8536689-e49e-431a-aafd-280839effc13 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.127133 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8536689-e49e-431a-aafd-280839effc13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.127133 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8536689-e49e-431a-aafd-280839effc13 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.127457 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8536689-e49e-431a-aafd-280839effc13 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.128219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8536689-e49e-431a-aafd-280839effc13 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.131873 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e221e62c-47d0-42aa-9af3-bb1c0d37a0e0 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.136717 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68b817e-351a-4a0a-ae64-3d20e76c05d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.137059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68b817e-351a-4a0a-ae64-3d20e76c05d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.137339 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68b817e-351a-4a0a-ae64-3d20e76c05d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.137613 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68b817e-351a-4a0a-ae64-3d20e76c05d7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.138235 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68b817e-351a-4a0a-ae64-3d20e76c05d7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.140179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68b817e-351a-4a0a-ae64-3d20e76c05d7 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.156174 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 780/1569] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:30 2022] POST /identity/v3/projects => generated 367 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:30.157628 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8536689-e49e-431a-aafd-280839effc13 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.171047 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 790/1570] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:30 2022] POST /identity/v3/projects => generated 373 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:30.172803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68b817e-351a-4a0a-ae64-3d20e76c05d7 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.226137 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8536689-e49e-431a-aafd-280839effc13 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.230766 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9385149a-8da9-4365-bf04-7b7bd82f0dff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.231500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9385149a-8da9-4365-bf04-7b7bd82f0dff admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.231779 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9385149a-8da9-4365-bf04-7b7bd82f0dff admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.232048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9385149a-8da9-4365-bf04-7b7bd82f0dff admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.232661 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9385149a-8da9-4365-bf04-7b7bd82f0dff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.234556 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9385149a-8da9-4365-bf04-7b7bd82f0dff admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.234687 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68b817e-351a-4a0a-ae64-3d20e76c05d7 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.239323 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b6f840-9d50-4ced-a3f7-7bf373b4bfd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.240021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b6f840-9d50-4ced-a3f7-7bf373b4bfd9 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.240233 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b6f840-9d50-4ced-a3f7-7bf373b4bfd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.240459 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b6f840-9d50-4ced-a3f7-7bf373b4bfd9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.241162 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b6f840-9d50-4ced-a3f7-7bf373b4bfd9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:18:30.241587 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b6f840-9d50-4ced-a3f7-7bf373b4bfd9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.243374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b6f840-9d50-4ced-a3f7-7bf373b4bfd9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.256665 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 781/1571] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:30 2022] POST /identity/v3/users => generated 296 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:30.264449 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 791/1572] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:18:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:18:30.264449 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b6f840-9d50-4ced-a3f7-7bf373b4bfd9 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.264715 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9385149a-8da9-4365-bf04-7b7bd82f0dff admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.276402 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b6f840-9d50-4ced-a3f7-7bf373b4bfd9 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.281394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86792f49-7d22-4089-a02a-d44ecb71bc2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.282501 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86792f49-7d22-4089-a02a-d44ecb71bc2b admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.282780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86792f49-7d22-4089-a02a-d44ecb71bc2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.283084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86792f49-7d22-4089-a02a-d44ecb71bc2b admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.283721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86792f49-7d22-4089-a02a-d44ecb71bc2b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:18:30.284021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86792f49-7d22-4089-a02a-d44ecb71bc2b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.285599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86792f49-7d22-4089-a02a-d44ecb71bc2b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.292908 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:30.299230 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 792/1573] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:18:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:18:30.300016 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86792f49-7d22-4089-a02a-d44ecb71bc2b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.310518 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86792f49-7d22-4089-a02a-d44ecb71bc2b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.315424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf8fbb4-2cfe-4282-8824-1b5a0c540123 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.316034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf8fbb4-2cfe-4282-8824-1b5a0c540123 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.316310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf8fbb4-2cfe-4282-8824-1b5a0c540123 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.316583 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf8fbb4-2cfe-4282-8824-1b5a0c540123 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.317198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf8fbb4-2cfe-4282-8824-1b5a0c540123 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.319074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf8fbb4-2cfe-4282-8824-1b5a0c540123 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.328534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9385149a-8da9-4365-bf04-7b7bd82f0dff admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.332264 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622ff761-19ca-4079-a13e-18ce7493a807 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.332904 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622ff761-19ca-4079-a13e-18ce7493a807 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.333185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622ff761-19ca-4079-a13e-18ce7493a807 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.333460 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622ff761-19ca-4079-a13e-18ce7493a807 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.344722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622ff761-19ca-4079-a13e-18ce7493a807 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.345977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 793/1574] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:30 2022] POST /identity/v3/projects => generated 387 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:30.346183 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622ff761-19ca-4079-a13e-18ce7493a807 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.346761 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf8fbb4-2cfe-4282-8824-1b5a0c540123 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.355962 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 782/1575] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:30.356736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622ff761-19ca-4079-a13e-18ce7493a807 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.366199 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:30.367167 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622ff761-19ca-4079-a13e-18ce7493a807 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.420321 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf8fbb4-2cfe-4282-8824-1b5a0c540123 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.425710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218696d1-212a-4918-8803-837c28ba06c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.426326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218696d1-212a-4918-8803-837c28ba06c0 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.426608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218696d1-212a-4918-8803-837c28ba06c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.426932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218696d1-212a-4918-8803-837c28ba06c0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.427566 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218696d1-212a-4918-8803-837c28ba06c0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.429509 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218696d1-212a-4918-8803-837c28ba06c0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.430392 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3476ec26-509e-4a82-b5f3-4c5ede6b7900 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.431122 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3476ec26-509e-4a82-b5f3-4c5ede6b7900 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.431467 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3476ec26-509e-4a82-b5f3-4c5ede6b7900 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.432076 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3476ec26-509e-4a82-b5f3-4c5ede6b7900 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.432578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3476ec26-509e-4a82-b5f3-4c5ede6b7900 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.434743 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3476ec26-509e-4a82-b5f3-4c5ede6b7900 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.460614 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 783/1576] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:30 2022] POST /identity/v3/users => generated 284 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:30.461744 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3476ec26-509e-4a82-b5f3-4c5ede6b7900 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.464195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 794/1577] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:30 2022] POST /identity/v3/projects => generated 377 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:30.465343 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218696d1-212a-4918-8803-837c28ba06c0 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.476268 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3476ec26-509e-4a82-b5f3-4c5ede6b7900 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.537390 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218696d1-212a-4918-8803-837c28ba06c0 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.542707 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c430126-3c06-4e87-8695-ac9f1f6af528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.543232 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1730a2f4-e8fa-4298-b601-d0645800326b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.543393 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c430126-3c06-4e87-8695-ac9f1f6af528 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.543614 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c430126-3c06-4e87-8695-ac9f1f6af528 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.543851 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1730a2f4-e8fa-4298-b601-d0645800326b admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.543967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c430126-3c06-4e87-8695-ac9f1f6af528 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.544358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1730a2f4-e8fa-4298-b601-d0645800326b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.544546 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c430126-3c06-4e87-8695-ac9f1f6af528 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.544677 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1730a2f4-e8fa-4298-b601-d0645800326b admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.545231 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1730a2f4-e8fa-4298-b601-d0645800326b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.546271 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c430126-3c06-4e87-8695-ac9f1f6af528 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.547727 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1730a2f4-e8fa-4298-b601-d0645800326b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.563400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 795/1578] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:30 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:30.564264 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c430126-3c06-4e87-8695-ac9f1f6af528 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.574957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 784/1579] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:30 2022] POST /identity/v3/users => generated 287 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:30.578365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c430126-3c06-4e87-8695-ac9f1f6af528 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.578522 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1730a2f4-e8fa-4298-b601-d0645800326b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.581871 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdbc205-c988-456c-8824-ecb7bcb5df5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.582934 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdbc205-c988-456c-8824-ecb7bcb5df5c admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.583226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdbc205-c988-456c-8824-ecb7bcb5df5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.583500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdbc205-c988-456c-8824-ecb7bcb5df5c admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.584132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdbc205-c988-456c-8824-ecb7bcb5df5c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.586023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdbc205-c988-456c-8824-ecb7bcb5df5c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.591643 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1730a2f4-e8fa-4298-b601-d0645800326b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.596752 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f179935-8bfb-4c0f-a973-f9d39337596c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.597440 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f179935-8bfb-4c0f-a973-f9d39337596c admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.597792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f179935-8bfb-4c0f-a973-f9d39337596c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.598136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f179935-8bfb-4c0f-a973-f9d39337596c admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.598899 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f179935-8bfb-4c0f-a973-f9d39337596c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.600930 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f179935-8bfb-4c0f-a973-f9d39337596c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.616908 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 796/1580] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:30 2022] POST /identity/v3/projects => generated 387 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:30.617829 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdbc205-c988-456c-8824-ecb7bcb5df5c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.636824 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 785/1581] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:30 2022] POST /identity/v3/projects => generated 373 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:30.637575 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f179935-8bfb-4c0f-a973-f9d39337596c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.667869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:30.699736 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdbc205-c988-456c-8824-ecb7bcb5df5c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.719746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f179935-8bfb-4c0f-a973-f9d39337596c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.724769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fcaedc-ddc6-4200-aed1-0bbd7ab45689 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.725560 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fcaedc-ddc6-4200-aed1-0bbd7ab45689 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.725849 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fcaedc-ddc6-4200-aed1-0bbd7ab45689 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.726128 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fcaedc-ddc6-4200-aed1-0bbd7ab45689 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.726776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fcaedc-ddc6-4200-aed1-0bbd7ab45689 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.728558 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fcaedc-ddc6-4200-aed1-0bbd7ab45689 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.729050 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:30.745515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 786/1582] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:30 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:30.746788 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fcaedc-ddc6-4200-aed1-0bbd7ab45689 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.759350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fcaedc-ddc6-4200-aed1-0bbd7ab45689 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.764456 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8c3074-7ab7-47f8-91a1-5360e42c24d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.765271 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8c3074-7ab7-47f8-91a1-5360e42c24d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.765558 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8c3074-7ab7-47f8-91a1-5360e42c24d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.766879 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8c3074-7ab7-47f8-91a1-5360e42c24d2 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.767587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8c3074-7ab7-47f8-91a1-5360e42c24d2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.769992 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8c3074-7ab7-47f8-91a1-5360e42c24d2 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.780517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eac481f-708a-4f24-a08f-6d480dbe6d48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.781138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eac481f-708a-4f24-a08f-6d480dbe6d48 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.781429 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eac481f-708a-4f24-a08f-6d480dbe6d48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.782272 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eac481f-708a-4f24-a08f-6d480dbe6d48 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.782942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eac481f-708a-4f24-a08f-6d480dbe6d48 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.783964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: WARNING py.warnings [None req-2eac481f-708a-4f24-a08f-6d480dbe6d48 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 01:18:30.783964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: warnings.warn(msg) Sep 13 01:18:30.783964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: Sep 13 01:18:30.785430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eac481f-708a-4f24-a08f-6d480dbe6d48 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.803820 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 787/1583] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:30 2022] POST /identity/v3/users => generated 289 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:30.803820 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8c3074-7ab7-47f8-91a1-5360e42c24d2 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.814287 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8c3074-7ab7-47f8-91a1-5360e42c24d2 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.815825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 797/1584] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:30 2022] POST /identity/v3/users => generated 294 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:30.820287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eac481f-708a-4f24-a08f-6d480dbe6d48 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.822219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97516fc6-3834-4b29-9f22-1f6c9a77152b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.822994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97516fc6-3834-4b29-9f22-1f6c9a77152b admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.823343 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97516fc6-3834-4b29-9f22-1f6c9a77152b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.823675 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97516fc6-3834-4b29-9f22-1f6c9a77152b admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.824379 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97516fc6-3834-4b29-9f22-1f6c9a77152b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.826052 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97516fc6-3834-4b29-9f22-1f6c9a77152b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.832853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eac481f-708a-4f24-a08f-6d480dbe6d48 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.838671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b89d36-8937-4315-a49d-99892efe567f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.839558 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b89d36-8937-4315-a49d-99892efe567f admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.839844 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b89d36-8937-4315-a49d-99892efe567f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.840155 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b89d36-8937-4315-a49d-99892efe567f admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.840834 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 788/1585] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:30 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:30.840970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b89d36-8937-4315-a49d-99892efe567f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.842517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b89d36-8937-4315-a49d-99892efe567f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.845422 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97516fc6-3834-4b29-9f22-1f6c9a77152b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.862002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 798/1586] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:30 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:30.862002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b89d36-8937-4315-a49d-99892efe567f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.862291 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97516fc6-3834-4b29-9f22-1f6c9a77152b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.865965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc74452-22c0-4a18-9c45-4b9742ca2e29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.866667 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc74452-22c0-4a18-9c45-4b9742ca2e29 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.867083 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc74452-22c0-4a18-9c45-4b9742ca2e29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.867361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc74452-22c0-4a18-9c45-4b9742ca2e29 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.868269 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc74452-22c0-4a18-9c45-4b9742ca2e29 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.870665 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc74452-22c0-4a18-9c45-4b9742ca2e29 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.879746 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b89d36-8937-4315-a49d-99892efe567f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.886402 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3154fd-ed7b-49ab-883b-ea58dd823620 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.887088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3154fd-ed7b-49ab-883b-ea58dd823620 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.887460 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3154fd-ed7b-49ab-883b-ea58dd823620 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.887753 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3154fd-ed7b-49ab-883b-ea58dd823620 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.888544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3154fd-ed7b-49ab-883b-ea58dd823620 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.890810 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3154fd-ed7b-49ab-883b-ea58dd823620 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.896313 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 789/1587] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:30 2022] POST /identity/v3/users => generated 294 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:30.897189 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc74452-22c0-4a18-9c45-4b9742ca2e29 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.909259 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc74452-22c0-4a18-9c45-4b9742ca2e29 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.914627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b178831-7cca-48ca-9a98-5d73bdf47cc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.915270 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b178831-7cca-48ca-9a98-5d73bdf47cc6 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.915549 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b178831-7cca-48ca-9a98-5d73bdf47cc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.915858 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b178831-7cca-48ca-9a98-5d73bdf47cc6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.916511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b178831-7cca-48ca-9a98-5d73bdf47cc6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.918153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b178831-7cca-48ca-9a98-5d73bdf47cc6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.919396 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 799/1588] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:30 2022] POST /identity/v3/users => generated 287 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:30.920244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3154fd-ed7b-49ab-883b-ea58dd823620 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.931634 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3154fd-ed7b-49ab-883b-ea58dd823620 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.932861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 790/1589] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:30 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:30.933777 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b178831-7cca-48ca-9a98-5d73bdf47cc6 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.941154 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061e47b3-3762-4fde-b2f1-90401e9d3abb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.941805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061e47b3-3762-4fde-b2f1-90401e9d3abb admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.942139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061e47b3-3762-4fde-b2f1-90401e9d3abb admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.942454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061e47b3-3762-4fde-b2f1-90401e9d3abb admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.943186 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061e47b3-3762-4fde-b2f1-90401e9d3abb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.945260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b178831-7cca-48ca-9a98-5d73bdf47cc6 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.945379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061e47b3-3762-4fde-b2f1-90401e9d3abb admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.949888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8198cb4a-a026-45bc-aa1e-1c57c278792d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.950488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8198cb4a-a026-45bc-aa1e-1c57c278792d admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.950766 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8198cb4a-a026-45bc-aa1e-1c57c278792d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.951218 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8198cb4a-a026-45bc-aa1e-1c57c278792d admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.951859 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8198cb4a-a026-45bc-aa1e-1c57c278792d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.954187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8198cb4a-a026-45bc-aa1e-1c57c278792d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.957070 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 800/1590] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:30 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:30.957942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061e47b3-3762-4fde-b2f1-90401e9d3abb admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.966651 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 791/1591] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:30 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:30.967658 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8198cb4a-a026-45bc-aa1e-1c57c278792d admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:30.970984 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061e47b3-3762-4fde-b2f1-90401e9d3abb admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.975683 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb423beb-95a8-4044-8eaf-608be61bc19e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.976352 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb423beb-95a8-4044-8eaf-608be61bc19e admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.976634 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb423beb-95a8-4044-8eaf-608be61bc19e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.976907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb423beb-95a8-4044-8eaf-608be61bc19e admin admin] PATH_INFO: `/v3/projects/a84dd36fe9484f41bc2f0d7da4e47c13/users/0c5dcfa5794843a5896fa051a4554e1f/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.978698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8198cb4a-a026-45bc-aa1e-1c57c278792d admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:30.983587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d1c118-410f-4c45-826f-ed4aa086417e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:30.984234 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d1c118-410f-4c45-826f-ed4aa086417e admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:30.984534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d1c118-410f-4c45-826f-ed4aa086417e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:30.984822 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d1c118-410f-4c45-826f-ed4aa086417e admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:30.985447 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d1c118-410f-4c45-826f-ed4aa086417e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.987081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d1c118-410f-4c45-826f-ed4aa086417e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:30.994544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb423beb-95a8-4044-8eaf-608be61bc19e admin admin] RBAC: Authorizing `identity:create_grant(project_id=a84dd36fe9484f41bc2f0d7da4e47c13, user_id=0c5dcfa5794843a5896fa051a4554e1f, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:30.997577 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb423beb-95a8-4044-8eaf-608be61bc19e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:31.001952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 792/1592] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:31.001952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d1c118-410f-4c45-826f-ed4aa086417e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:31.014551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d1c118-410f-4c45-826f-ed4aa086417e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:31.018985 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 801/1593] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:30 2022] PUT /identity/v3/projects/a84dd36fe9484f41bc2f0d7da4e47c13/users/0c5dcfa5794843a5896fa051a4554e1f/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:31.020156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb423beb-95a8-4044-8eaf-608be61bc19e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:31.088549 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2038263b-2e6d-438f-a6d0-aa21c4c6744b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:31.089475 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2038263b-2e6d-438f-a6d0-aa21c4c6744b admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:31.089475 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2038263b-2e6d-438f-a6d0-aa21c4c6744b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:31.089691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2038263b-2e6d-438f-a6d0-aa21c4c6744b admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:31.090692 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2038263b-2e6d-438f-a6d0-aa21c4c6744b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:31.092110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2038263b-2e6d-438f-a6d0-aa21c4c6744b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:31.105419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 793/1594] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:30 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:31.106538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2038263b-2e6d-438f-a6d0-aa21c4c6744b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:31.118421 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2038263b-2e6d-438f-a6d0-aa21c4c6744b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:31.125238 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e09a4b-3079-44da-804d-32cc14420901 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:31.126888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e09a4b-3079-44da-804d-32cc14420901 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:31.127192 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e09a4b-3079-44da-804d-32cc14420901 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:31.128051 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e09a4b-3079-44da-804d-32cc14420901 admin admin] PATH_INFO: `/v3/projects/d09aabd4960043f5ae0efae870db28fb/users/21815172a4ea4eca8cf5b67415e3b1ff/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:31.131882 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb423beb-95a8-4044-8eaf-608be61bc19e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:31.138097 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e877946-767c-4411-9ac1-5a8ad979fded admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:31.138996 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e877946-767c-4411-9ac1-5a8ad979fded admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:31.139574 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e877946-767c-4411-9ac1-5a8ad979fded admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:31.140092 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e877946-767c-4411-9ac1-5a8ad979fded admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:31.141454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e877946-767c-4411-9ac1-5a8ad979fded admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:31.143689 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e877946-767c-4411-9ac1-5a8ad979fded admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:31.151758 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e09a4b-3079-44da-804d-32cc14420901 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d09aabd4960043f5ae0efae870db28fb, user_id=21815172a4ea4eca8cf5b67415e3b1ff, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:31.156409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e09a4b-3079-44da-804d-32cc14420901 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:31.161499 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 802/1595] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:31 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:31.162563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e877946-767c-4411-9ac1-5a8ad979fded admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:31.174153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e877946-767c-4411-9ac1-5a8ad979fded admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:31.180000 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 794/1596] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:31 2022] PUT /identity/v3/projects/d09aabd4960043f5ae0efae870db28fb/users/21815172a4ea4eca8cf5b67415e3b1ff/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:31.183723 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e09a4b-3079-44da-804d-32cc14420901 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:31.209543 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:31.239879 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:31.271456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adb4895-ebf2-472c-8bae-f0d882cf67a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:31.272118 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adb4895-ebf2-472c-8bae-f0d882cf67a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:31.272397 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adb4895-ebf2-472c-8bae-f0d882cf67a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:31.272724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adb4895-ebf2-472c-8bae-f0d882cf67a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:31.273429 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6adb4895-ebf2-472c-8bae-f0d882cf67a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:31.275334 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6adb4895-ebf2-472c-8bae-f0d882cf67a7 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:31.287730 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 803/1597] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:31 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:31.288769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adb4895-ebf2-472c-8bae-f0d882cf67a7 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:31.289389 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e09a4b-3079-44da-804d-32cc14420901 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:31.295332 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81a2d80-e239-4ae4-8ef4-b26afec8ee83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:31.295977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81a2d80-e239-4ae4-8ef4-b26afec8ee83 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:31.296277 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81a2d80-e239-4ae4-8ef4-b26afec8ee83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:31.296534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81a2d80-e239-4ae4-8ef4-b26afec8ee83 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:31.297375 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81a2d80-e239-4ae4-8ef4-b26afec8ee83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:31.299063 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81a2d80-e239-4ae4-8ef4-b26afec8ee83 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:31.300442 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adb4895-ebf2-472c-8bae-f0d882cf67a7 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:31.305717 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a4017d-6192-4a07-b6b2-67e9243db531 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:31.306679 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a4017d-6192-4a07-b6b2-67e9243db531 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:31.307257 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a4017d-6192-4a07-b6b2-67e9243db531 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:31.307733 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a4017d-6192-4a07-b6b2-67e9243db531 admin admin] PATH_INFO: `/v3/projects/768dedda6202441c89b6d64055c98913/users/d7ae54329f314168a85a05672d3a5fdc/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:31.317532 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 795/1598] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:31 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:31.320300 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de68964c-4333-480a-94b8-aea6e78de374 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:31.320872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de68964c-4333-480a-94b8-aea6e78de374 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:31.321314 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de68964c-4333-480a-94b8-aea6e78de374 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:31.338774 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a4017d-6192-4a07-b6b2-67e9243db531 admin admin] RBAC: Authorizing `identity:create_grant(project_id=768dedda6202441c89b6d64055c98913, user_id=d7ae54329f314168a85a05672d3a5fdc, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:31.343665 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a4017d-6192-4a07-b6b2-67e9243db531 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:31.344968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-de68964c-4333-480a-94b8-aea6e78de374 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:31.346476 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-de68964c-4333-480a-94b8-aea6e78de374 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:31.361341 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 796/1599] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:31.361341 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de68964c-4333-480a-94b8-aea6e78de374 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:31.368047 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 804/1600] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:31 2022] PUT /identity/v3/projects/768dedda6202441c89b6d64055c98913/users/d7ae54329f314168a85a05672d3a5fdc/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:31.369004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a4017d-6192-4a07-b6b2-67e9243db531 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:31.378414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de68964c-4333-480a-94b8-aea6e78de374 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:31.451155 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a4017d-6192-4a07-b6b2-67e9243db531 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:31.457280 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe58369-c35f-4f0d-b1a1-bfccc81b3469 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:31.458033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe58369-c35f-4f0d-b1a1-bfccc81b3469 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:31.458322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe58369-c35f-4f0d-b1a1-bfccc81b3469 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:31.458599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe58369-c35f-4f0d-b1a1-bfccc81b3469 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:31.459283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe58369-c35f-4f0d-b1a1-bfccc81b3469 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:31.461427 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe58369-c35f-4f0d-b1a1-bfccc81b3469 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:31.470960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1517e0-66cd-4d19-9179-82a1fa0ce373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:31.471700 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1517e0-66cd-4d19-9179-82a1fa0ce373 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:31.472104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1517e0-66cd-4d19-9179-82a1fa0ce373 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:31.474894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1517e0-66cd-4d19-9179-82a1fa0ce373 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:31.475045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 805/1601] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:31 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:31.475146 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1517e0-66cd-4d19-9179-82a1fa0ce373 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:31.475338 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec66fc33-64da-41ef-80b6-8332b64c0210 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:31.475338 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec66fc33-64da-41ef-80b6-8332b64c0210 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:31.475338 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec66fc33-64da-41ef-80b6-8332b64c0210 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:31.475596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1517e0-66cd-4d19-9179-82a1fa0ce373 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:31.493591 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 797/1602] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:31 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:31.495313 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1517e0-66cd-4d19-9179-82a1fa0ce373 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:31.507023 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1517e0-66cd-4d19-9179-82a1fa0ce373 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:31.512610 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5f35bf-37c6-4e1e-83b4-12f3a42c7703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:31.513579 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5f35bf-37c6-4e1e-83b4-12f3a42c7703 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:31.513862 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5f35bf-37c6-4e1e-83b4-12f3a42c7703 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:31.514140 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5f35bf-37c6-4e1e-83b4-12f3a42c7703 admin admin] PATH_INFO: `/v3/projects/6475b1b1e54c4b0eb86021d82021de78/users/06797d4f56954444b3317f2339f020c4/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:31.514916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ec66fc33-64da-41ef-80b6-8332b64c0210 None None] MFA Rules not processed for user `21815172a4ea4eca8cf5b67415e3b1ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:31.516686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ec66fc33-64da-41ef-80b6-8332b64c0210 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:31.534427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5f35bf-37c6-4e1e-83b4-12f3a42c7703 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6475b1b1e54c4b0eb86021d82021de78, user_id=06797d4f56954444b3317f2339f020c4, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:31.538365 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5f35bf-37c6-4e1e-83b4-12f3a42c7703 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:31.558949 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 798/1603] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:31 2022] PUT /identity/v3/projects/6475b1b1e54c4b0eb86021d82021de78/users/06797d4f56954444b3317f2339f020c4/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:31.561646 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5f35bf-37c6-4e1e-83b4-12f3a42c7703 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:31.623712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 806/1604] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:31 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:31.624655 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec66fc33-64da-41ef-80b6-8332b64c0210 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:31.626597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ec66fc33-64da-41ef-80b6-8332b64c0210 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:31.652693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5f35bf-37c6-4e1e-83b4-12f3a42c7703 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:31.660573 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031c0625-d19d-4c82-966f-a09339ea9eb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:31.661736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031c0625-d19d-4c82-966f-a09339ea9eb4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:31.662499 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031c0625-d19d-4c82-966f-a09339ea9eb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:31.663259 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031c0625-d19d-4c82-966f-a09339ea9eb4 admin admin] PATH_INFO: `/v3/projects/9350ad0ccbc54f05bba32baa1229faef/users/79d2ef866c4f4fed83b85c4ee3192dda/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:31.681605 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031c0625-d19d-4c82-966f-a09339ea9eb4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9350ad0ccbc54f05bba32baa1229faef, user_id=79d2ef866c4f4fed83b85c4ee3192dda, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:31.686577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031c0625-d19d-4c82-966f-a09339ea9eb4 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:31.700822 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:31.711471 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 799/1605] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:31 2022] PUT /identity/v3/projects/9350ad0ccbc54f05bba32baa1229faef/users/79d2ef866c4f4fed83b85c4ee3192dda/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:31.713275 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d57dbf-0473-4414-8f99-2158ada7aac2 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:31.713762 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d57dbf-0473-4414-8f99-2158ada7aac2 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:31.714448 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d57dbf-0473-4414-8f99-2158ada7aac2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:31.716914 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:31.733570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec66fc33-64da-41ef-80b6-8332b64c0210 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:31.751026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-64d57dbf-0473-4414-8f99-2158ada7aac2 None None] MFA Rules not processed for user `d7ae54329f314168a85a05672d3a5fdc`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:31.752279 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-64d57dbf-0473-4414-8f99-2158ada7aac2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:31.807650 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5d9575-14bb-4842-8b22-83120fee6fd8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:31.808758 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5d9575-14bb-4842-8b22-83120fee6fd8 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:31.809294 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5d9575-14bb-4842-8b22-83120fee6fd8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:31.809782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5d9575-14bb-4842-8b22-83120fee6fd8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:31.835614 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 800/1606] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:31 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:31.836566 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d57dbf-0473-4414-8f99-2158ada7aac2 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:31.900730 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5d9575-14bb-4842-8b22-83120fee6fd8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:31.903423 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5d9575-14bb-4842-8b22-83120fee6fd8 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:31.921509 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 807/1607] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:31.922414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5d9575-14bb-4842-8b22-83120fee6fd8 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:31.934675 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5d9575-14bb-4842-8b22-83120fee6fd8 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:31.945099 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638624d9-d9e2-4ed0-939b-7b982bc15c88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:31.945944 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638624d9-d9e2-4ed0-939b-7b982bc15c88 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:31.946236 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638624d9-d9e2-4ed0-939b-7b982bc15c88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:31.946516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638624d9-d9e2-4ed0-939b-7b982bc15c88 admin admin] PATH_INFO: `/v3/projects/7a9e1eee32f04f3cabd8219d36d2c6a7/users/e2846ad486314958bb74af71b647c470/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:31.989972 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638624d9-d9e2-4ed0-939b-7b982bc15c88 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a9e1eee32f04f3cabd8219d36d2c6a7, user_id=e2846ad486314958bb74af71b647c470, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:31.997374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d57dbf-0473-4414-8f99-2158ada7aac2 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:31.999482 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638624d9-d9e2-4ed0-939b-7b982bc15c88 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:32.006434 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaf3d5a-991f-430d-85e2-88feb3b7b219 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:32.007168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaf3d5a-991f-430d-85e2-88feb3b7b219 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:32.007502 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaf3d5a-991f-430d-85e2-88feb3b7b219 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:32.007775 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaf3d5a-991f-430d-85e2-88feb3b7b219 admin admin] PATH_INFO: `/v3/projects/303be6708ab546b8a5d2c8ed0d525523/users/6c95627524fb413e9b087232c59cd818/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:32.021732 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 808/1608] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:31 2022] PUT /identity/v3/projects/7a9e1eee32f04f3cabd8219d36d2c6a7/users/e2846ad486314958bb74af71b647c470/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:32.024527 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d9ec3f-782f-45ff-869a-0c7d8becb9a8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:32.024527 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d9ec3f-782f-45ff-869a-0c7d8becb9a8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:32.024527 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d9ec3f-782f-45ff-869a-0c7d8becb9a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:32.040091 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaf3d5a-991f-430d-85e2-88feb3b7b219 admin admin] RBAC: Authorizing `identity:create_grant(project_id=303be6708ab546b8a5d2c8ed0d525523, user_id=6c95627524fb413e9b087232c59cd818, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:32.042824 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaf3d5a-991f-430d-85e2-88feb3b7b219 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:32.053563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-94d9ec3f-782f-45ff-869a-0c7d8becb9a8 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:32.054301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-94d9ec3f-782f-45ff-869a-0c7d8becb9a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:32.100948 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 801/1609] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:31 2022] PUT /identity/v3/projects/303be6708ab546b8a5d2c8ed0d525523/users/6c95627524fb413e9b087232c59cd818/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:32.101880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaf3d5a-991f-430d-85e2-88feb3b7b219 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:32.136259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 809/1610] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:32.137788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4dc8d37-add0-4291-b666-bff01efcef91 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:32.138110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4dc8d37-add0-4291-b666-bff01efcef91 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:32.138478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4dc8d37-add0-4291-b666-bff01efcef91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:32.180285 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b4dc8d37-add0-4291-b666-bff01efcef91 None None] MFA Rules not processed for user `79d2ef866c4f4fed83b85c4ee3192dda`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:32.182069 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b4dc8d37-add0-4291-b666-bff01efcef91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:32.209157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:32.230823 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaf3d5a-991f-430d-85e2-88feb3b7b219 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:32.238469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c42745-0fb6-4593-8348-c75ffd173292 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:32.239284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c42745-0fb6-4593-8348-c75ffd173292 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:32.240495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c42745-0fb6-4593-8348-c75ffd173292 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:32.240844 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c42745-0fb6-4593-8348-c75ffd173292 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:32.358608 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:32.358835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 810/1611] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:32 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:32.358835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6cf95c-093a-444e-82d2-2f67a932e76b None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:32.358835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6cf95c-093a-444e-82d2-2f67a932e76b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:32.358835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6cf95c-093a-444e-82d2-2f67a932e76b None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:32.358835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 811/1612] 10.222.0.15 () {54 vars in 962 bytes} [Tue Sep 13 01:18:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:18:32.358835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b6b32b-6fed-427c-b65e-0942f2110c9c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:32.358835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b6b32b-6fed-427c-b65e-0942f2110c9c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:32.360399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b6b32b-6fed-427c-b65e-0942f2110c9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:32.373806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c42745-0fb6-4593-8348-c75ffd173292 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:32.377474 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c42745-0fb6-4593-8348-c75ffd173292 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:32.389584 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-69b6b32b-6fed-427c-b65e-0942f2110c9c None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:32.391748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-69b6b32b-6fed-427c-b65e-0942f2110c9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:32.409690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 802/1613] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:32 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:32.413073 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9983b036-e412-442a-926b-a74e80429b0b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:32.413362 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9983b036-e412-442a-926b-a74e80429b0b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:32.413635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9983b036-e412-442a-926b-a74e80429b0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:32.508854 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9983b036-e412-442a-926b-a74e80429b0b None None] MFA Rules not processed for user `6c95627524fb413e9b087232c59cd818`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:32.509958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9983b036-e412-442a-926b-a74e80429b0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:32.544633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 812/1614] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:32.546128 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b6b32b-6fed-427c-b65e-0942f2110c9c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:32.637986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b6b32b-6fed-427c-b65e-0942f2110c9c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:32.643553 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae434f1b-4109-4b27-a06e-8802b12c863c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:32.644240 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae434f1b-4109-4b27-a06e-8802b12c863c service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:32.644633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae434f1b-4109-4b27-a06e-8802b12c863c service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:32.645132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae434f1b-4109-4b27-a06e-8802b12c863c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:32.685574 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae434f1b-4109-4b27-a06e-8802b12c863c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:32.691943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae434f1b-4109-4b27-a06e-8802b12c863c service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:32.724865 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 813/1615] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:32.728327 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 803/1616] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:32 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 318 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:32.730137 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae434f1b-4109-4b27-a06e-8802b12c863c service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:32.731160 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7fcd30-b32c-4ef7-8dc1-071c49a5a72a None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:32.731454 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7fcd30-b32c-4ef7-8dc1-071c49a5a72a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:32.733692 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7fcd30-b32c-4ef7-8dc1-071c49a5a72a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:32.756108 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae434f1b-4109-4b27-a06e-8802b12c863c service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:32.764227 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb5abe5-0f45-4d16-921d-24e37bb8efcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:32.767484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb5abe5-0f45-4d16-921d-24e37bb8efcd admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:32.767911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb5abe5-0f45-4d16-921d-24e37bb8efcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:32.768211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb5abe5-0f45-4d16-921d-24e37bb8efcd admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:32.768886 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb5abe5-0f45-4d16-921d-24e37bb8efcd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:32.771180 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb5abe5-0f45-4d16-921d-24e37bb8efcd admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:32.819353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 814/1617] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:32 2022] POST /identity/v3/projects => generated 385 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:32.820216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb5abe5-0f45-4d16-921d-24e37bb8efcd admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:32.835079 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9a7fcd30-b32c-4ef7-8dc1-071c49a5a72a None None] MFA Rules not processed for user `2f2da4d2d3604d16b58dc25b8c17c49a`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:32.839244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9a7fcd30-b32c-4ef7-8dc1-071c49a5a72a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:32.901208 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:32.949353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb5abe5-0f45-4d16-921d-24e37bb8efcd admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:32.972336 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb59b8a-7b5e-4259-9e05-134057f2dda9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:32.973682 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb59b8a-7b5e-4259-9e05-134057f2dda9 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:32.974374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb59b8a-7b5e-4259-9e05-134057f2dda9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:32.974832 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb59b8a-7b5e-4259-9e05-134057f2dda9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:33.014327 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 804/1618] 10.222.0.15 () {58 vars in 1059 bytes} [Tue Sep 13 01:18:32 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 285 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:33.015464 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7fcd30-b32c-4ef7-8dc1-071c49a5a72a None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:33.095273 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:33.131760 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb59b8a-7b5e-4259-9e05-134057f2dda9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:33.133309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb59b8a-7b5e-4259-9e05-134057f2dda9 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:33.166533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 815/1619] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:33.166923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7fcd30-b32c-4ef7-8dc1-071c49a5a72a None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:33.167523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb59b8a-7b5e-4259-9e05-134057f2dda9 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:33.175133 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a75e2e-14a7-401e-adca-2b9b403b5475 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:33.180070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a75e2e-14a7-401e-adca-2b9b403b5475 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:33.181131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a75e2e-14a7-401e-adca-2b9b403b5475 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:33.181294 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a75e2e-14a7-401e-adca-2b9b403b5475 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:33.182029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a75e2e-14a7-401e-adca-2b9b403b5475 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:33.184189 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a75e2e-14a7-401e-adca-2b9b403b5475 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:33.193147 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb59b8a-7b5e-4259-9e05-134057f2dda9 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:33.203318 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5480723f-82da-405b-b24c-7af93d7d4fd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:33.204929 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5480723f-82da-405b-b24c-7af93d7d4fd7 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:33.205265 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5480723f-82da-405b-b24c-7af93d7d4fd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:33.205546 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5480723f-82da-405b-b24c-7af93d7d4fd7 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:33.206190 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5480723f-82da-405b-b24c-7af93d7d4fd7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:33.209089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5480723f-82da-405b-b24c-7af93d7d4fd7 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:33.245175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 805/1620] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:33 2022] POST /identity/v3/projects => generated 373 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:33.254096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a75e2e-14a7-401e-adca-2b9b403b5475 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:33.272861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 816/1621] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:33 2022] POST /identity/v3/users => generated 299 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:33.275195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5480723f-82da-405b-b24c-7af93d7d4fd7 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:33.306697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5480723f-82da-405b-b24c-7af93d7d4fd7 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:33.443268 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d08f1ec-b493-4e31-934e-79a3e5464ed9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:33.445810 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d08f1ec-b493-4e31-934e-79a3e5464ed9 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:33.445970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d08f1ec-b493-4e31-934e-79a3e5464ed9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:33.446658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d08f1ec-b493-4e31-934e-79a3e5464ed9 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:33.448091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d08f1ec-b493-4e31-934e-79a3e5464ed9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:33.454956 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d08f1ec-b493-4e31-934e-79a3e5464ed9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:33.483388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a75e2e-14a7-401e-adca-2b9b403b5475 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:33.501819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2841a098-ea47-49a2-86b4-9fc2561e2ff9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:33.502716 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2841a098-ea47-49a2-86b4-9fc2561e2ff9 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:33.503135 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2841a098-ea47-49a2-86b4-9fc2561e2ff9 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:33.503645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2841a098-ea47-49a2-86b4-9fc2561e2ff9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:33.516904 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 817/1622] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:33 2022] POST /identity/v3/users => generated 293 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:33.517858 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d08f1ec-b493-4e31-934e-79a3e5464ed9 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:33.568349 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d08f1ec-b493-4e31-934e-79a3e5464ed9 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:33.598904 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5a624c-d046-4b01-b974-df2b546b84c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:33.599559 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5a624c-d046-4b01-b974-df2b546b84c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:33.599845 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5a624c-d046-4b01-b974-df2b546b84c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:33.600118 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5a624c-d046-4b01-b974-df2b546b84c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:33.600786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5a624c-d046-4b01-b974-df2b546b84c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:33.602404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5a624c-d046-4b01-b974-df2b546b84c7 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:33.642561 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 818/1623] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:33 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:33.643295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5a624c-d046-4b01-b974-df2b546b84c7 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:33.672818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2841a098-ea47-49a2-86b4-9fc2561e2ff9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:33.674313 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5a624c-d046-4b01-b974-df2b546b84c7 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:33.674843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2841a098-ea47-49a2-86b4-9fc2561e2ff9 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:33.690840 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02573fe0-e6cc-456f-831a-9c912f229f28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:33.691539 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02573fe0-e6cc-456f-831a-9c912f229f28 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:33.691774 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02573fe0-e6cc-456f-831a-9c912f229f28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:33.692068 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02573fe0-e6cc-456f-831a-9c912f229f28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:33.692740 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02573fe0-e6cc-456f-831a-9c912f229f28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:33.694547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02573fe0-e6cc-456f-831a-9c912f229f28 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:33.697013 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 806/1624] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:33 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 451 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:33.700003 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2841a098-ea47-49a2-86b4-9fc2561e2ff9 service nova] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:33.728621 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 819/1625] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:33 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:33.732827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02573fe0-e6cc-456f-831a-9c912f229f28 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:33.742431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2841a098-ea47-49a2-86b4-9fc2561e2ff9 service nova] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:33.751700 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02573fe0-e6cc-456f-831a-9c912f229f28 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:33.756283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e70d7b-5f80-49ee-9c74-efa3e5e7d887 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:33.757135 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e70d7b-5f80-49ee-9c74-efa3e5e7d887 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:33.757498 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e70d7b-5f80-49ee-9c74-efa3e5e7d887 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:33.757887 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e70d7b-5f80-49ee-9c74-efa3e5e7d887 admin admin] PATH_INFO: `/v3/projects/b9ad81ba0cdc4e88ba78ba44b495bdc5/users/41039927184a4b589f8784c75dafedb4/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:33.763495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ded724-824d-4909-9755-4378ae4a1d6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:33.764186 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ded724-824d-4909-9755-4378ae4a1d6c admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:33.764467 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ded724-824d-4909-9755-4378ae4a1d6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:33.764744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ded724-824d-4909-9755-4378ae4a1d6c admin admin] PATH_INFO: `/v3/projects/5e92e5aa357c4b2aa13612e0ba07313d/users/c3657954be884cad935e6c62eed4fe99/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:33.794900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e70d7b-5f80-49ee-9c74-efa3e5e7d887 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b9ad81ba0cdc4e88ba78ba44b495bdc5, user_id=41039927184a4b589f8784c75dafedb4, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:33.798245 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ded724-824d-4909-9755-4378ae4a1d6c admin admin] RBAC: Authorizing `identity:create_grant(project_id=5e92e5aa357c4b2aa13612e0ba07313d, user_id=c3657954be884cad935e6c62eed4fe99, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:33.801113 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ded724-824d-4909-9755-4378ae4a1d6c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:33.801486 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e70d7b-5f80-49ee-9c74-efa3e5e7d887 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:33.830962 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 820/1626] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:33 2022] PUT /identity/v3/projects/5e92e5aa357c4b2aa13612e0ba07313d/users/c3657954be884cad935e6c62eed4fe99/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:18:33.834780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ded724-824d-4909-9755-4378ae4a1d6c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:33.844101 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 807/1627] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:33 2022] PUT /identity/v3/projects/b9ad81ba0cdc4e88ba78ba44b495bdc5/users/41039927184a4b589f8784c75dafedb4/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:33.848117 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e70d7b-5f80-49ee-9c74-efa3e5e7d887 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:33.860414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:33.932719 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:33.997206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e70d7b-5f80-49ee-9c74-efa3e5e7d887 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:34.012006 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72786726-2644-458a-980b-afd9f489a95b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:34.012802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72786726-2644-458a-980b-afd9f489a95b admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:34.013126 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72786726-2644-458a-980b-afd9f489a95b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:34.013492 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72786726-2644-458a-980b-afd9f489a95b admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:34.014262 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72786726-2644-458a-980b-afd9f489a95b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:34.016046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72786726-2644-458a-980b-afd9f489a95b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:34.022153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ded724-824d-4909-9755-4378ae4a1d6c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:34.043578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 808/1628] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:33 2022] GET /identity/v3/roles => generated 1420 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:18:34.044711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72786726-2644-458a-980b-afd9f489a95b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:34.051776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545bf998-cc0f-4b1c-9bad-cd6dc0ce5a39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:34.052337 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545bf998-cc0f-4b1c-9bad-cd6dc0ce5a39 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:34.052696 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545bf998-cc0f-4b1c-9bad-cd6dc0ce5a39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:34.052976 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545bf998-cc0f-4b1c-9bad-cd6dc0ce5a39 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:34.053649 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545bf998-cc0f-4b1c-9bad-cd6dc0ce5a39 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:34.057265 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72786726-2644-458a-980b-afd9f489a95b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:34.057787 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545bf998-cc0f-4b1c-9bad-cd6dc0ce5a39 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:34.069701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb6b461-4489-4a4c-be42-35c12f388ce7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:34.070734 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb6b461-4489-4a4c-be42-35c12f388ce7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:34.071257 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb6b461-4489-4a4c-be42-35c12f388ce7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:34.071752 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb6b461-4489-4a4c-be42-35c12f388ce7 admin admin] PATH_INFO: `/v3/projects/b9ad81ba0cdc4e88ba78ba44b495bdc5/users/41039927184a4b589f8784c75dafedb4/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:34.078676 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb6b461-4489-4a4c-be42-35c12f388ce7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b9ad81ba0cdc4e88ba78ba44b495bdc5, user_id=41039927184a4b589f8784c75dafedb4, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:34.082833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb6b461-4489-4a4c-be42-35c12f388ce7 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:34.087571 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 821/1629] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:33 2022] GET /identity/v3/roles => generated 1420 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:34.088744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545bf998-cc0f-4b1c-9bad-cd6dc0ce5a39 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:34.113228 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 809/1630] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:34 2022] PUT /identity/v3/projects/b9ad81ba0cdc4e88ba78ba44b495bdc5/users/41039927184a4b589f8784c75dafedb4/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:34.122535 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545bf998-cc0f-4b1c-9bad-cd6dc0ce5a39 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:34.124533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761fd8ed-147d-44b8-8ef8-a40a048a8ca7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:34.126199 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761fd8ed-147d-44b8-8ef8-a40a048a8ca7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:34.127193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761fd8ed-147d-44b8-8ef8-a40a048a8ca7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:34.164680 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-761fd8ed-147d-44b8-8ef8-a40a048a8ca7 None None] MFA Rules not processed for user `41039927184a4b589f8784c75dafedb4`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:34.166679 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-761fd8ed-147d-44b8-8ef8-a40a048a8ca7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:34.263390 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 810/1631] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:34 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:34.271670 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761fd8ed-147d-44b8-8ef8-a40a048a8ca7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:34.305352 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e258a72-cdfa-4ab8-ab1c-7c6a0846fbd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:34.306187 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e258a72-cdfa-4ab8-ab1c-7c6a0846fbd8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:34.306473 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e258a72-cdfa-4ab8-ab1c-7c6a0846fbd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:34.306752 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e258a72-cdfa-4ab8-ab1c-7c6a0846fbd8 admin admin] PATH_INFO: `/v3/projects/5e92e5aa357c4b2aa13612e0ba07313d/users/c3657954be884cad935e6c62eed4fe99/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:34.314646 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e258a72-cdfa-4ab8-ab1c-7c6a0846fbd8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5e92e5aa357c4b2aa13612e0ba07313d, user_id=c3657954be884cad935e6c62eed4fe99, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:34.336850 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e258a72-cdfa-4ab8-ab1c-7c6a0846fbd8 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:34.373133 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 822/1632] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:34 2022] PUT /identity/v3/projects/5e92e5aa357c4b2aa13612e0ba07313d/users/c3657954be884cad935e6c62eed4fe99/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:34.381438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746464fd-e9f9-411f-9605-f5bedb660c9a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:34.381722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746464fd-e9f9-411f-9605-f5bedb660c9a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:34.382000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746464fd-e9f9-411f-9605-f5bedb660c9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:34.405380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761fd8ed-147d-44b8-8ef8-a40a048a8ca7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:34.473024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-746464fd-e9f9-411f-9605-f5bedb660c9a None None] MFA Rules not processed for user `c3657954be884cad935e6c62eed4fe99`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:34.474318 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-746464fd-e9f9-411f-9605-f5bedb660c9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:34.477194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5d3dc1-a98e-4b47-a79a-81a331e9be9d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:34.478780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5d3dc1-a98e-4b47-a79a-81a331e9be9d service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:34.479235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5d3dc1-a98e-4b47-a79a-81a331e9be9d service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:34.479713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5d3dc1-a98e-4b47-a79a-81a331e9be9d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:34.498246 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:34.660261 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5d3dc1-a98e-4b47-a79a-81a331e9be9d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:34.662170 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5d3dc1-a98e-4b47-a79a-81a331e9be9d service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:34.666597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 823/1633] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:34 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 286 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:18:34.675498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e51936e-d7ab-4e28-aea0-a802d9cb3c5c None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:34.677506 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e51936e-d7ab-4e28-aea0-a802d9cb3c5c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:34.677876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e51936e-d7ab-4e28-aea0-a802d9cb3c5c None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:34.682954 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 824/1634] 10.222.0.15 () {54 vars in 912 bytes} [Tue Sep 13 01:18:34 2022] GET /identity => generated 270 bytes in 16 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:18:34.686716 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e51936e-d7ab-4e28-aea0-a802d9cb3c5c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:34.701093 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 811/1635] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:34 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:18:34.705603 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff323a36-4673-4aaf-84e7-8b30dc8d16f4 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:34.707227 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff323a36-4673-4aaf-84e7-8b30dc8d16f4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:34.708980 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff323a36-4673-4aaf-84e7-8b30dc8d16f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:34.718798 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e51936e-d7ab-4e28-aea0-a802d9cb3c5c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:34.734325 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f679d3-a1dd-49c1-afe0-06cf53161702 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:34.736494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f679d3-a1dd-49c1-afe0-06cf53161702 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:34.740144 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f679d3-a1dd-49c1-afe0-06cf53161702 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:34.742890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f679d3-a1dd-49c1-afe0-06cf53161702 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:34.788209 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f679d3-a1dd-49c1-afe0-06cf53161702 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:34.795091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f679d3-a1dd-49c1-afe0-06cf53161702 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:34.821017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ff323a36-4673-4aaf-84e7-8b30dc8d16f4 None None] MFA Rules not processed for user `2f2da4d2d3604d16b58dc25b8c17c49a`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:34.822490 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ff323a36-4673-4aaf-84e7-8b30dc8d16f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:34.823385 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 825/1636] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:34 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:34.824255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f679d3-a1dd-49c1-afe0-06cf53161702 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:34.869823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:35.020170 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 812/1637] 10.222.0.15 () {58 vars in 1009 bytes} [Tue Sep 13 01:18:34 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 318 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:35.041232 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff323a36-4673-4aaf-84e7-8b30dc8d16f4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:35.056144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff323a36-4673-4aaf-84e7-8b30dc8d16f4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:35.057589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f679d3-a1dd-49c1-afe0-06cf53161702 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:35.071743 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569bb2f4-e56e-4b2e-934f-5e175ac14b12 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:35.072254 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569bb2f4-e56e-4b2e-934f-5e175ac14b12 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:35.072535 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569bb2f4-e56e-4b2e-934f-5e175ac14b12 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:35.072806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569bb2f4-e56e-4b2e-934f-5e175ac14b12 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:35.075686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba31c852-9cdb-4dbc-be2d-18b4429de84d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:35.078042 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba31c852-9cdb-4dbc-be2d-18b4429de84d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:35.080658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba31c852-9cdb-4dbc-be2d-18b4429de84d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:35.080947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba31c852-9cdb-4dbc-be2d-18b4429de84d admin admin] PATH_INFO: `/v3/users/79d2ef866c4f4fed83b85c4ee3192dda` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:35.083588 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba31c852-9cdb-4dbc-be2d-18b4429de84d admin admin] RBAC: Authorizing `identity:delete_user(user_id=79d2ef866c4f4fed83b85c4ee3192dda)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:35.089710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba31c852-9cdb-4dbc-be2d-18b4429de84d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:35.111111 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569bb2f4-e56e-4b2e-934f-5e175ac14b12 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:35.121797 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569bb2f4-e56e-4b2e-934f-5e175ac14b12 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:35.162031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 813/1638] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:35.211663 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ba31c852-9cdb-4dbc-be2d-18b4429de84d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '79d2ef866c4f4fed83b85c4ee3192dda'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:35.212471 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ba31c852-9cdb-4dbc-be2d-18b4429de84d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '79d2ef866c4f4fed83b85c4ee3192dda'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:35.228620 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ba31c852-9cdb-4dbc-be2d-18b4429de84d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '79d2ef866c4f4fed83b85c4ee3192dda'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:35.229956 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ba31c852-9cdb-4dbc-be2d-18b4429de84d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '79d2ef866c4f4fed83b85c4ee3192dda'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:35.257635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ba31c852-9cdb-4dbc-be2d-18b4429de84d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '79d2ef866c4f4fed83b85c4ee3192dda'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:35.292019 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 826/1639] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:18:34 2022] DELETE /identity/v3/users/79d2ef866c4f4fed83b85c4ee3192dda => generated 0 bytes in 468 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:35.297911 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1ddfd2-54cd-4248-a8ac-c18f3bdbc05b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:35.298380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1ddfd2-54cd-4248-a8ac-c18f3bdbc05b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:35.298475 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1ddfd2-54cd-4248-a8ac-c18f3bdbc05b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:35.347518 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ea1ddfd2-54cd-4248-a8ac-c18f3bdbc05b None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:35.350142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ea1ddfd2-54cd-4248-a8ac-c18f3bdbc05b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:35.556777 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 814/1640] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 260 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:35.773482 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba31c852-9cdb-4dbc-be2d-18b4429de84d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:35.774966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ba31c852-9cdb-4dbc-be2d-18b4429de84d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:35.790244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1ddfd2-54cd-4248-a8ac-c18f3bdbc05b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:35.792800 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ea1ddfd2-54cd-4248-a8ac-c18f3bdbc05b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:35.798317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba31c852-9cdb-4dbc-be2d-18b4429de84d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:35.814211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33fbea0-1845-4887-9c2e-2613c30d64a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:35.814963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33fbea0-1845-4887-9c2e-2613c30d64a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:35.815244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33fbea0-1845-4887-9c2e-2613c30d64a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:35.815514 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33fbea0-1845-4887-9c2e-2613c30d64a2 admin admin] PATH_INFO: `/v3/users/6c95627524fb413e9b087232c59cd818` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:35.818901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33fbea0-1845-4887-9c2e-2613c30d64a2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6c95627524fb413e9b087232c59cd818)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:35.821029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33fbea0-1845-4887-9c2e-2613c30d64a2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:35.877983 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:35.931329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f33fbea0-1845-4887-9c2e-2613c30d64a2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6c95627524fb413e9b087232c59cd818'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:35.932361 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f33fbea0-1845-4887-9c2e-2613c30d64a2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6c95627524fb413e9b087232c59cd818'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:35.947448 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f33fbea0-1845-4887-9c2e-2613c30d64a2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6c95627524fb413e9b087232c59cd818'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:35.948950 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f33fbea0-1845-4887-9c2e-2613c30d64a2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6c95627524fb413e9b087232c59cd818'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:35.970701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1ddfd2-54cd-4248-a8ac-c18f3bdbc05b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:35.984416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e7db96-0283-431b-8eee-5b0a789a8b37 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:35.986681 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e7db96-0283-431b-8eee-5b0a789a8b37 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:35.987002 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e7db96-0283-431b-8eee-5b0a789a8b37 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:35.987284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e7db96-0283-431b-8eee-5b0a789a8b37 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:35.990023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f33fbea0-1845-4887-9c2e-2613c30d64a2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6c95627524fb413e9b087232c59cd818'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:35.992319 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-66e7db96-0283-431b-8eee-5b0a789a8b37 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:36.019470 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 827/1641] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:18:35 2022] DELETE /identity/v3/users/6c95627524fb413e9b087232c59cd818 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:36.029659 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d796d0-3e11-4c9d-91f1-d8ab3c63b7a3 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:36.030406 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d796d0-3e11-4c9d-91f1-d8ab3c63b7a3 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:36.030794 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e7db96-0283-431b-8eee-5b0a789a8b37 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:36.032624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e7db96-0283-431b-8eee-5b0a789a8b37 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:36.033175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d796d0-3e11-4c9d-91f1-d8ab3c63b7a3 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:36.037194 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 828/1642] 10.222.0.15 () {54 vars in 962 bytes} [Tue Sep 13 01:18:36 2022] GET /identity => generated 270 bytes in 17 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:18:36.038808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a989e7d1-9ed0-4013-906c-c5de6d56841e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:36.043046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a989e7d1-9ed0-4013-906c-c5de6d56841e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:36.043644 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a989e7d1-9ed0-4013-906c-c5de6d56841e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:36.059039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 815/1643] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 269 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:36.060860 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd2552e-81d9-47a7-a12a-3da23d656a6f None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:36.061750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd2552e-81d9-47a7-a12a-3da23d656a6f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:36.062238 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd2552e-81d9-47a7-a12a-3da23d656a6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:36.064499 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:36.092013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a989e7d1-9ed0-4013-906c-c5de6d56841e None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:36.093225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a989e7d1-9ed0-4013-906c-c5de6d56841e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:36.096533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-5cd2552e-81d9-47a7-a12a-3da23d656a6f None None] MFA Rules not processed for user `2f2da4d2d3604d16b58dc25b8c17c49a`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:36.097789 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5cd2552e-81d9-47a7-a12a-3da23d656a6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:36.350289 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 816/1644] 10.222.0.15 () {58 vars in 1059 bytes} [Tue Sep 13 01:18:36 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 291 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:36.357794 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f613e15-3c84-4f13-92c9-9a12895fcf08 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:36.358258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f613e15-3c84-4f13-92c9-9a12895fcf08 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:36.358664 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f613e15-3c84-4f13-92c9-9a12895fcf08 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:36.361038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 817/1645] 10.222.0.15 () {54 vars in 912 bytes} [Tue Sep 13 01:18:36 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:18:36.367639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7002ef-ac04-4f20-9233-5d66badfe245 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:36.368061 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7002ef-ac04-4f20-9233-5d66badfe245 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:36.368489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7002ef-ac04-4f20-9233-5d66badfe245 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:36.375039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 829/1646] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 337 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:36.420263 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a989e7d1-9ed0-4013-906c-c5de6d56841e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:36.420416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-3b7002ef-ac04-4f20-9233-5d66badfe245 None None] MFA Rules not processed for user `2f2da4d2d3604d16b58dc25b8c17c49a`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:36.421619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3b7002ef-ac04-4f20-9233-5d66badfe245 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:36.423015 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a989e7d1-9ed0-4013-906c-c5de6d56841e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:36.443696 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 818/1647] 10.222.0.15 () {58 vars in 1009 bytes} [Tue Sep 13 01:18:36 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:36.454789 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7002ef-ac04-4f20-9233-5d66badfe245 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:36.456239 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3b7002ef-ac04-4f20-9233-5d66badfe245 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:36.581625 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a989e7d1-9ed0-4013-906c-c5de6d56841e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:36.591067 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e859744-b974-440e-a9ad-e5f96b1b1549 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:36.591638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7002ef-ac04-4f20-9233-5d66badfe245 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:36.591762 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e859744-b974-440e-a9ad-e5f96b1b1549 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:36.592226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e859744-b974-440e-a9ad-e5f96b1b1549 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:36.592505 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e859744-b974-440e-a9ad-e5f96b1b1549 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:36.609956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0479ae2d-1058-4797-a0a4-9d8b4a3d4ef3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:36.612296 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0479ae2d-1058-4797-a0a4-9d8b4a3d4ef3 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:36.612671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0479ae2d-1058-4797-a0a4-9d8b4a3d4ef3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:36.613493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0479ae2d-1058-4797-a0a4-9d8b4a3d4ef3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:36.623554 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e859744-b974-440e-a9ad-e5f96b1b1549 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:36.626303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e859744-b974-440e-a9ad-e5f96b1b1549 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:36.655081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0479ae2d-1058-4797-a0a4-9d8b4a3d4ef3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:36.670547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 830/1648] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:18:36.672986 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0479ae2d-1058-4797-a0a4-9d8b4a3d4ef3 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:36.706315 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 819/1649] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:36 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 252 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:18:38.153537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e859744-b974-440e-a9ad-e5f96b1b1549 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:38.156692 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5e859744-b974-440e-a9ad-e5f96b1b1549 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:38.163537 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0479ae2d-1058-4797-a0a4-9d8b4a3d4ef3 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:38.165284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0479ae2d-1058-4797-a0a4-9d8b4a3d4ef3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:38.180813 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e859744-b974-440e-a9ad-e5f96b1b1549 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:38.196962 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705225e3-dd75-4a7f-b1b5-22ee1917f75e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:38.197849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705225e3-dd75-4a7f-b1b5-22ee1917f75e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:38.199376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705225e3-dd75-4a7f-b1b5-22ee1917f75e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:38.199808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705225e3-dd75-4a7f-b1b5-22ee1917f75e admin admin] PATH_INFO: `/v3/projects/9350ad0ccbc54f05bba32baa1229faef` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:38.201322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705225e3-dd75-4a7f-b1b5-22ee1917f75e admin admin] RBAC: Authorizing `identity:delete_project(project_id=9350ad0ccbc54f05bba32baa1229faef)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:38.203450 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0479ae2d-1058-4797-a0a4-9d8b4a3d4ef3 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:38.208575 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0a71bb-03c3-4ef3-9a14-685aa5749731 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:38.209490 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0a71bb-03c3-4ef3-9a14-685aa5749731 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:38.209965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0a71bb-03c3-4ef3-9a14-685aa5749731 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:38.215031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0a71bb-03c3-4ef3-9a14-685aa5749731 admin admin] PATH_INFO: `/v3/projects/303be6708ab546b8a5d2c8ed0d525523` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:38.215163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: WARNING py.warnings [None req-705225e3-dd75-4a7f-b1b5-22ee1917f75e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 01:18:38.215163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: warnings.warn(msg) Sep 13 01:18:38.215163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: Sep 13 01:18:38.216861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705225e3-dd75-4a7f-b1b5-22ee1917f75e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:38.217411 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0a71bb-03c3-4ef3-9a14-685aa5749731 admin admin] RBAC: Authorizing `identity:delete_project(project_id=303be6708ab546b8a5d2c8ed0d525523)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:38.219526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0a71bb-03c3-4ef3-9a14-685aa5749731 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:38.338947 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7e0a71bb-03c3-4ef3-9a14-685aa5749731 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '303be6708ab546b8a5d2c8ed0d525523', 'request_id': 'req-7e0a71bb-03c3-4ef3-9a14-685aa5749731'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.341374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:38.357043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7e0a71bb-03c3-4ef3-9a14-685aa5749731 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '303be6708ab546b8a5d2c8ed0d525523', 'request_id': 'req-7e0a71bb-03c3-4ef3-9a14-685aa5749731'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.364142 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-705225e3-dd75-4a7f-b1b5-22ee1917f75e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9350ad0ccbc54f05bba32baa1229faef', 'request_id': 'req-705225e3-dd75-4a7f-b1b5-22ee1917f75e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.394175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-705225e3-dd75-4a7f-b1b5-22ee1917f75e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9350ad0ccbc54f05bba32baa1229faef', 'request_id': 'req-705225e3-dd75-4a7f-b1b5-22ee1917f75e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.397124 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7e0a71bb-03c3-4ef3-9a14-685aa5749731 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '303be6708ab546b8a5d2c8ed0d525523', 'request_id': 'req-7e0a71bb-03c3-4ef3-9a14-685aa5749731'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.424446 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7e0a71bb-03c3-4ef3-9a14-685aa5749731 admin admin] The token cache is being invalidate because project 303be6708ab546b8a5d2c8ed0d525523 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:18:38.424833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7e0a71bb-03c3-4ef3-9a14-685aa5749731 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.426032 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7e0a71bb-03c3-4ef3-9a14-685aa5749731 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.426718 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-705225e3-dd75-4a7f-b1b5-22ee1917f75e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9350ad0ccbc54f05bba32baa1229faef', 'request_id': 'req-705225e3-dd75-4a7f-b1b5-22ee1917f75e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.428733 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 820/1650] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:18:38 2022] DELETE /identity/v3/projects/303be6708ab546b8a5d2c8ed0d525523 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:38.432555 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0a71bb-03c3-4ef3-9a14-685aa5749731 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:38.435433 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7e0a71bb-03c3-4ef3-9a14-685aa5749731 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:38.455770 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-705225e3-dd75-4a7f-b1b5-22ee1917f75e admin admin] The token cache is being invalidate because project 9350ad0ccbc54f05bba32baa1229faef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:18:38.457839 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-705225e3-dd75-4a7f-b1b5-22ee1917f75e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.459121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-705225e3-dd75-4a7f-b1b5-22ee1917f75e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.473876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 831/1651] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:18:38 2022] DELETE /identity/v3/projects/9350ad0ccbc54f05bba32baa1229faef => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:38.474903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705225e3-dd75-4a7f-b1b5-22ee1917f75e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:38.484040 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-705225e3-dd75-4a7f-b1b5-22ee1917f75e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:38.596795 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:38.632010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0a71bb-03c3-4ef3-9a14-685aa5749731 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:38.644276 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292945a9-48c5-4181-b2db-00418da68991 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:38.644819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292945a9-48c5-4181-b2db-00418da68991 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:38.646918 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292945a9-48c5-4181-b2db-00418da68991 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:38.648228 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292945a9-48c5-4181-b2db-00418da68991 admin admin] PATH_INFO: `/v3/users/c3657954be884cad935e6c62eed4fe99` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:38.651525 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292945a9-48c5-4181-b2db-00418da68991 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c3657954be884cad935e6c62eed4fe99)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:38.655303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: WARNING py.warnings [None req-292945a9-48c5-4181-b2db-00418da68991 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 01:18:38.655303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: warnings.warn(msg) Sep 13 01:18:38.655303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: Sep 13 01:18:38.661443 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292945a9-48c5-4181-b2db-00418da68991 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:38.670988 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705225e3-dd75-4a7f-b1b5-22ee1917f75e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:38.701864 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2b8287-2c4d-471d-a5ef-76ff60b98563 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:38.703498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2b8287-2c4d-471d-a5ef-76ff60b98563 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:38.705973 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2b8287-2c4d-471d-a5ef-76ff60b98563 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:38.706302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2b8287-2c4d-471d-a5ef-76ff60b98563 admin admin] PATH_INFO: `/v3/users/41039927184a4b589f8784c75dafedb4` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:38.708190 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2b8287-2c4d-471d-a5ef-76ff60b98563 admin admin] RBAC: Authorizing `identity:delete_user(user_id=41039927184a4b589f8784c75dafedb4)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:38.710138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2b8287-2c4d-471d-a5ef-76ff60b98563 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:38.846200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7f2b8287-2c4d-471d-a5ef-76ff60b98563 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '41039927184a4b589f8784c75dafedb4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.847110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7f2b8287-2c4d-471d-a5ef-76ff60b98563 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '41039927184a4b589f8784c75dafedb4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.853573 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-292945a9-48c5-4181-b2db-00418da68991 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c3657954be884cad935e6c62eed4fe99'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.855812 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-292945a9-48c5-4181-b2db-00418da68991 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c3657954be884cad935e6c62eed4fe99'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.868278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7f2b8287-2c4d-471d-a5ef-76ff60b98563 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '41039927184a4b589f8784c75dafedb4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.868951 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7f2b8287-2c4d-471d-a5ef-76ff60b98563 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '41039927184a4b589f8784c75dafedb4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.877350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-292945a9-48c5-4181-b2db-00418da68991 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c3657954be884cad935e6c62eed4fe99'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.877607 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-292945a9-48c5-4181-b2db-00418da68991 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c3657954be884cad935e6c62eed4fe99'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.904330 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-292945a9-48c5-4181-b2db-00418da68991 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c3657954be884cad935e6c62eed4fe99'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.906292 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-7f2b8287-2c4d-471d-a5ef-76ff60b98563 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '41039927184a4b589f8784c75dafedb4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:38.925169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 832/1652] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:18:38 2022] DELETE /identity/v3/users/41039927184a4b589f8784c75dafedb4 => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:38.950145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 821/1653] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:18:38 2022] DELETE /identity/v3/users/c3657954be884cad935e6c62eed4fe99 => generated 0 bytes in 515 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:18:39.167692 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2b8287-2c4d-471d-a5ef-76ff60b98563 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:39.169851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7f2b8287-2c4d-471d-a5ef-76ff60b98563 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:39.193208 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9a253f-df46-41b6-a6cb-1e381670ed42 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:39.193714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9a253f-df46-41b6-a6cb-1e381670ed42 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:39.193714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9a253f-df46-41b6-a6cb-1e381670ed42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:39.262443 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-fd9a253f-df46-41b6-a6cb-1e381670ed42 None None] MFA Rules not processed for user `349e19300f1e48c3ac44fc34611c7f4a`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:39.265273 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fd9a253f-df46-41b6-a6cb-1e381670ed42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:39.355613 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2b8287-2c4d-471d-a5ef-76ff60b98563 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:39.446394 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:39.508973 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 822/1654] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:39 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 317 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:18:39.525748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:39.543694 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9a253f-df46-41b6-a6cb-1e381670ed42 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:39.547658 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fd9a253f-df46-41b6-a6cb-1e381670ed42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:39.569187 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772c7a2b-a315-4b7e-b284-5251a1adf658 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:39.585889 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772c7a2b-a315-4b7e-b284-5251a1adf658 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:39.589341 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772c7a2b-a315-4b7e-b284-5251a1adf658 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:39.590233 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772c7a2b-a315-4b7e-b284-5251a1adf658 admin admin] PATH_INFO: `/v3/users/d7ae54329f314168a85a05672d3a5fdc` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:39.596196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772c7a2b-a315-4b7e-b284-5251a1adf658 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d7ae54329f314168a85a05672d3a5fdc)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:39.603503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772c7a2b-a315-4b7e-b284-5251a1adf658 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:39.702981 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-772c7a2b-a315-4b7e-b284-5251a1adf658 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd7ae54329f314168a85a05672d3a5fdc'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:39.706972 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-772c7a2b-a315-4b7e-b284-5251a1adf658 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd7ae54329f314168a85a05672d3a5fdc'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:39.725541 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-772c7a2b-a315-4b7e-b284-5251a1adf658 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd7ae54329f314168a85a05672d3a5fdc'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:39.726600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-772c7a2b-a315-4b7e-b284-5251a1adf658 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd7ae54329f314168a85a05672d3a5fdc'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:39.765187 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-772c7a2b-a315-4b7e-b284-5251a1adf658 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd7ae54329f314168a85a05672d3a5fdc'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:39.789216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9a253f-df46-41b6-a6cb-1e381670ed42 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:39.800626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 833/1655] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:18:39 2022] DELETE /identity/v3/users/d7ae54329f314168a85a05672d3a5fdc => generated 0 bytes in 633 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:39.804287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791b3e40-3cfc-4735-b0b7-53e32da7fecc None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:39.804646 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791b3e40-3cfc-4735-b0b7-53e32da7fecc None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:39.805061 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5dea4d-8590-434e-b501-2c91a09c5c62 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:39.805693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5dea4d-8590-434e-b501-2c91a09c5c62 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:39.805968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5dea4d-8590-434e-b501-2c91a09c5c62 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:39.806233 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5dea4d-8590-434e-b501-2c91a09c5c62 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:39.806917 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791b3e40-3cfc-4735-b0b7-53e32da7fecc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:39.811947 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ab5dea4d-8590-434e-b501-2c91a09c5c62 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:39.847042 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5dea4d-8590-434e-b501-2c91a09c5c62 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:39.852133 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5dea4d-8590-434e-b501-2c91a09c5c62 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:39.866100 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-791b3e40-3cfc-4735-b0b7-53e32da7fecc None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:39.868189 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-791b3e40-3cfc-4735-b0b7-53e32da7fecc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:39.904287 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 823/1656] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:39 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:40.091383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 834/1657] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 285 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:40.111315 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5dea4d-8590-434e-b501-2c91a09c5c62 service nova] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:40.113392 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ab5dea4d-8590-434e-b501-2c91a09c5c62 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:40.216157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5dea4d-8590-434e-b501-2c91a09c5c62 service nova] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:40.307763 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12414cc-9dfd-451b-bd79-b98a9dde6f94 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:40.309397 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12414cc-9dfd-451b-bd79-b98a9dde6f94 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:40.309737 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12414cc-9dfd-451b-bd79-b98a9dde6f94 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:40.310021 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12414cc-9dfd-451b-bd79-b98a9dde6f94 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:40.333365 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12414cc-9dfd-451b-bd79-b98a9dde6f94 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:40.336827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12414cc-9dfd-451b-bd79-b98a9dde6f94 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:40.350980 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 824/1658] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:40.353008 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791b3e40-3cfc-4735-b0b7-53e32da7fecc None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:40.353625 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-791b3e40-3cfc-4735-b0b7-53e32da7fecc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:40.381536 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791b3e40-3cfc-4735-b0b7-53e32da7fecc None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:40.394495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b175034b-9c71-4cd3-b629-1d1d5b79d7ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:40.395459 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b175034b-9c71-4cd3-b629-1d1d5b79d7ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:40.405807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b175034b-9c71-4cd3-b629-1d1d5b79d7ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:40.407084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b175034b-9c71-4cd3-b629-1d1d5b79d7ac admin admin] PATH_INFO: `/v3/projects/5e92e5aa357c4b2aa13612e0ba07313d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:40.411000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b175034b-9c71-4cd3-b629-1d1d5b79d7ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=5e92e5aa357c4b2aa13612e0ba07313d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:40.412958 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b175034b-9c71-4cd3-b629-1d1d5b79d7ac admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:40.489228 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12414cc-9dfd-451b-bd79-b98a9dde6f94 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:40.491113 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d12414cc-9dfd-451b-bd79-b98a9dde6f94 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:40.517420 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12414cc-9dfd-451b-bd79-b98a9dde6f94 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:40.540937 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b175034b-9c71-4cd3-b629-1d1d5b79d7ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5e92e5aa357c4b2aa13612e0ba07313d', 'request_id': 'req-b175034b-9c71-4cd3-b629-1d1d5b79d7ac'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:40.553989 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b175034b-9c71-4cd3-b629-1d1d5b79d7ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5e92e5aa357c4b2aa13612e0ba07313d', 'request_id': 'req-b175034b-9c71-4cd3-b629-1d1d5b79d7ac'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:40.569650 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b175034b-9c71-4cd3-b629-1d1d5b79d7ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5e92e5aa357c4b2aa13612e0ba07313d', 'request_id': 'req-b175034b-9c71-4cd3-b629-1d1d5b79d7ac'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:40.587890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:40.595674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b175034b-9c71-4cd3-b629-1d1d5b79d7ac admin admin] The token cache is being invalidate because project 5e92e5aa357c4b2aa13612e0ba07313d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:18:40.596229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b175034b-9c71-4cd3-b629-1d1d5b79d7ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:40.598736 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b175034b-9c71-4cd3-b629-1d1d5b79d7ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:40.605051 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:40.607443 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 835/1659] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:18:40 2022] DELETE /identity/v3/projects/5e92e5aa357c4b2aa13612e0ba07313d => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:40.608942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c47407-046d-47be-b54e-c4f35feadd3e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:40.609226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c47407-046d-47be-b54e-c4f35feadd3e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:40.616292 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c47407-046d-47be-b54e-c4f35feadd3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:40.667152 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-00c47407-046d-47be-b54e-c4f35feadd3e None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:40.668657 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-00c47407-046d-47be-b54e-c4f35feadd3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:40.674476 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ae020a-b70d-4330-aa3b-e7cc8f16c4ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:40.675171 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ae020a-b70d-4330-aa3b-e7cc8f16c4ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:40.675453 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ae020a-b70d-4330-aa3b-e7cc8f16c4ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:40.675726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ae020a-b70d-4330-aa3b-e7cc8f16c4ec admin admin] PATH_INFO: `/v3/projects/b9ad81ba0cdc4e88ba78ba44b495bdc5` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:40.678173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ae020a-b70d-4330-aa3b-e7cc8f16c4ec admin admin] RBAC: Authorizing `identity:delete_project(project_id=b9ad81ba0cdc4e88ba78ba44b495bdc5)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:40.679776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ae020a-b70d-4330-aa3b-e7cc8f16c4ec admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:40.683309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 836/1660] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:40.696585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6c1157-199f-4ab8-a21e-57e8244ff127 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:40.696585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6c1157-199f-4ab8-a21e-57e8244ff127 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:40.696712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6c1157-199f-4ab8-a21e-57e8244ff127 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:40.749414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-0c6c1157-199f-4ab8-a21e-57e8244ff127 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:40.753358 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0c6c1157-199f-4ab8-a21e-57e8244ff127 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:40.792451 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-69ae020a-b70d-4330-aa3b-e7cc8f16c4ec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b9ad81ba0cdc4e88ba78ba44b495bdc5', 'request_id': 'req-69ae020a-b70d-4330-aa3b-e7cc8f16c4ec'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:40.807510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-69ae020a-b70d-4330-aa3b-e7cc8f16c4ec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b9ad81ba0cdc4e88ba78ba44b495bdc5', 'request_id': 'req-69ae020a-b70d-4330-aa3b-e7cc8f16c4ec'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:40.819217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-69ae020a-b70d-4330-aa3b-e7cc8f16c4ec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b9ad81ba0cdc4e88ba78ba44b495bdc5', 'request_id': 'req-69ae020a-b70d-4330-aa3b-e7cc8f16c4ec'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:40.843204 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-69ae020a-b70d-4330-aa3b-e7cc8f16c4ec admin admin] The token cache is being invalidate because project b9ad81ba0cdc4e88ba78ba44b495bdc5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:18:40.843510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-69ae020a-b70d-4330-aa3b-e7cc8f16c4ec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:40.845169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-69ae020a-b70d-4330-aa3b-e7cc8f16c4ec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:40.853492 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 825/1661] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:18:40 2022] DELETE /identity/v3/projects/b9ad81ba0cdc4e88ba78ba44b495bdc5 => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:40.857857 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fb2fa6-6178-444c-a508-d0fdab26e4c3 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:40.858107 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fb2fa6-6178-444c-a508-d0fdab26e4c3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:40.866904 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fb2fa6-6178-444c-a508-d0fdab26e4c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:40.922635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 837/1662] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:40.924162 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6c1157-199f-4ab8-a21e-57e8244ff127 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:40.926637 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-31fb2fa6-6178-444c-a508-d0fdab26e4c3 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:40.928342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-31fb2fa6-6178-444c-a508-d0fdab26e4c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:40.941887 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 826/1663] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:40.960688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b768c4e-6434-497f-8f70-275b5f948c50 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:40.960886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b768c4e-6434-497f-8f70-275b5f948c50 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:40.961170 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b768c4e-6434-497f-8f70-275b5f948c50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:40.986093 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6c1157-199f-4ab8-a21e-57e8244ff127 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:41.006882 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05443c08-b985-4cf8-afbb-dcca59382300 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:41.006882 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05443c08-b985-4cf8-afbb-dcca59382300 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:41.006882 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05443c08-b985-4cf8-afbb-dcca59382300 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:41.008758 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05443c08-b985-4cf8-afbb-dcca59382300 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:41.008868 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05443c08-b985-4cf8-afbb-dcca59382300 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:18:41.009784 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05443c08-b985-4cf8-afbb-dcca59382300 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:41.012878 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05443c08-b985-4cf8-afbb-dcca59382300 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:41.033705 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-0b768c4e-6434-497f-8f70-275b5f948c50 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:41.034954 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0b768c4e-6434-497f-8f70-275b5f948c50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:41.060415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 838/1664] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:18:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:18:41.063387 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05443c08-b985-4cf8-afbb-dcca59382300 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:41.065712 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 827/1665] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:41.072144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b768c4e-6434-497f-8f70-275b5f948c50 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:41.083190 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05443c08-b985-4cf8-afbb-dcca59382300 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:41.096648 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f290098-3d43-4ee7-80d1-efc2bab20b2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:41.097108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b768c4e-6434-497f-8f70-275b5f948c50 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:41.097755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f290098-3d43-4ee7-80d1-efc2bab20b2e admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:41.097755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f290098-3d43-4ee7-80d1-efc2bab20b2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:41.097909 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f290098-3d43-4ee7-80d1-efc2bab20b2e admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:41.098601 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f290098-3d43-4ee7-80d1-efc2bab20b2e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:41.102191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f290098-3d43-4ee7-80d1-efc2bab20b2e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:41.111381 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cbf3d7-965c-4c7d-bc89-dbc1ae820786 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:41.111847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cbf3d7-965c-4c7d-bc89-dbc1ae820786 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:41.114691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cbf3d7-965c-4c7d-bc89-dbc1ae820786 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:41.114691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cbf3d7-965c-4c7d-bc89-dbc1ae820786 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:41.117965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cbf3d7-965c-4c7d-bc89-dbc1ae820786 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:18:41.118306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cbf3d7-965c-4c7d-bc89-dbc1ae820786 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:41.138635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cbf3d7-965c-4c7d-bc89-dbc1ae820786 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:41.168836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 839/1666] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:41 2022] POST /identity/v3/projects => generated 361 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:41.170554 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f290098-3d43-4ee7-80d1-efc2bab20b2e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:41.195880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 828/1667] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:18:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:18:41.195880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cbf3d7-965c-4c7d-bc89-dbc1ae820786 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:41.224287 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cbf3d7-965c-4c7d-bc89-dbc1ae820786 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:41.305407 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f290098-3d43-4ee7-80d1-efc2bab20b2e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:41.324738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1570a3ee-0fd7-4f87-9088-cb093c280b2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:41.325298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1570a3ee-0fd7-4f87-9088-cb093c280b2c admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:41.325585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1570a3ee-0fd7-4f87-9088-cb093c280b2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:41.325968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1570a3ee-0fd7-4f87-9088-cb093c280b2c admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:41.332150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1570a3ee-0fd7-4f87-9088-cb093c280b2c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:41.332909 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aca3d58-b302-41d4-980a-ebcd11f76e8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:41.334052 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aca3d58-b302-41d4-980a-ebcd11f76e8c admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:41.334540 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aca3d58-b302-41d4-980a-ebcd11f76e8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:41.335150 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aca3d58-b302-41d4-980a-ebcd11f76e8c admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:41.336647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aca3d58-b302-41d4-980a-ebcd11f76e8c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:41.337387 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1570a3ee-0fd7-4f87-9088-cb093c280b2c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:41.340425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aca3d58-b302-41d4-980a-ebcd11f76e8c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:41.393739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 840/1668] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:41 2022] POST /identity/v3/users => generated 281 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:41.421237 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1570a3ee-0fd7-4f87-9088-cb093c280b2c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:41.429048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 829/1669] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:41 2022] POST /identity/v3/projects => generated 381 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:41.431203 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aca3d58-b302-41d4-980a-ebcd11f76e8c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:41.451652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1570a3ee-0fd7-4f87-9088-cb093c280b2c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:41.496610 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:41.523560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:41.541700 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf8cc2c-2276-49bc-b930-733e8ec30a95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:41.542333 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf8cc2c-2276-49bc-b930-733e8ec30a95 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:41.542609 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf8cc2c-2276-49bc-b930-733e8ec30a95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:41.542923 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf8cc2c-2276-49bc-b930-733e8ec30a95 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:41.543584 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf8cc2c-2276-49bc-b930-733e8ec30a95 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:41.545466 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf8cc2c-2276-49bc-b930-733e8ec30a95 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:41.557123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 841/1670] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:41 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:41.560897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf8cc2c-2276-49bc-b930-733e8ec30a95 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:41.572766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf8cc2c-2276-49bc-b930-733e8ec30a95 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:41.578113 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728154c0-081e-42dc-9f8f-2d2e5b62f9de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:41.578995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728154c0-081e-42dc-9f8f-2d2e5b62f9de admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:41.579378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728154c0-081e-42dc-9f8f-2d2e5b62f9de admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:41.579775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728154c0-081e-42dc-9f8f-2d2e5b62f9de admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:41.580780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728154c0-081e-42dc-9f8f-2d2e5b62f9de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:41.582343 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728154c0-081e-42dc-9f8f-2d2e5b62f9de admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:41.596539 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 842/1671] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:41 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:18:41.601943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728154c0-081e-42dc-9f8f-2d2e5b62f9de admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:41.627502 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aca3d58-b302-41d4-980a-ebcd11f76e8c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:41.627877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728154c0-081e-42dc-9f8f-2d2e5b62f9de admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:41.652718 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe73d37-3745-4541-aa2c-89d52e267e2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:41.654268 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b2f394-66bb-4d50-b897-9ab6b49465ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:41.655311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe73d37-3745-4541-aa2c-89d52e267e2b admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:41.655701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe73d37-3745-4541-aa2c-89d52e267e2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:41.656102 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe73d37-3745-4541-aa2c-89d52e267e2b admin admin] PATH_INFO: `/v3/projects/2031a924ca5c4eb9919e291aea2178bb/users/31ce4daf5ef64741ba5b4dd363453096/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:41.657157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b2f394-66bb-4d50-b897-9ab6b49465ec admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:41.657525 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b2f394-66bb-4d50-b897-9ab6b49465ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:41.657812 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b2f394-66bb-4d50-b897-9ab6b49465ec admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:41.658572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b2f394-66bb-4d50-b897-9ab6b49465ec admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:41.663295 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b2f394-66bb-4d50-b897-9ab6b49465ec admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:41.708067 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe73d37-3745-4541-aa2c-89d52e267e2b admin admin] RBAC: Authorizing `identity:create_grant(project_id=2031a924ca5c4eb9919e291aea2178bb, user_id=31ce4daf5ef64741ba5b4dd363453096, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:41.710614 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 830/1672] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:41 2022] POST /identity/v3/users => generated 291 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:41.717795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b2f394-66bb-4d50-b897-9ab6b49465ec admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:41.718540 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe73d37-3745-4541-aa2c-89d52e267e2b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:41.737774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b2f394-66bb-4d50-b897-9ab6b49465ec admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:41.751260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326a69d9-93fb-48d4-9e99-960fbe6b632c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:41.752713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326a69d9-93fb-48d4-9e99-960fbe6b632c admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:41.753268 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326a69d9-93fb-48d4-9e99-960fbe6b632c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:41.753929 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326a69d9-93fb-48d4-9e99-960fbe6b632c admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:41.755020 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326a69d9-93fb-48d4-9e99-960fbe6b632c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:41.757576 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326a69d9-93fb-48d4-9e99-960fbe6b632c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:41.785521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 843/1673] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:41 2022] PUT /identity/v3/projects/2031a924ca5c4eb9919e291aea2178bb/users/31ce4daf5ef64741ba5b4dd363453096/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:41.794200 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 831/1674] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:41 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:18:41.795930 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08e432d-479d-49a6-b7f0-bdc23533fe54 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:41.796320 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08e432d-479d-49a6-b7f0-bdc23533fe54 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:41.796765 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08e432d-479d-49a6-b7f0-bdc23533fe54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:41.803274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326a69d9-93fb-48d4-9e99-960fbe6b632c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:41.818838 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326a69d9-93fb-48d4-9e99-960fbe6b632c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:41.873274 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a08e432d-479d-49a6-b7f0-bdc23533fe54 None None] MFA Rules not processed for user `31ce4daf5ef64741ba5b4dd363453096`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:41.874602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a08e432d-479d-49a6-b7f0-bdc23533fe54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:41.991976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c276251-9e7f-469f-b0e2-601e606737c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:41.992609 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c276251-9e7f-469f-b0e2-601e606737c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:41.992894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c276251-9e7f-469f-b0e2-601e606737c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:41.993166 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c276251-9e7f-469f-b0e2-601e606737c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:41.993814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c276251-9e7f-469f-b0e2-601e606737c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:41.995447 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c276251-9e7f-469f-b0e2-601e606737c8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:42.038631 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 832/1675] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:41 2022] GET /identity/v3/roles => generated 1420 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:18:42.044274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c276251-9e7f-469f-b0e2-601e606737c8 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:42.080436 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c276251-9e7f-469f-b0e2-601e606737c8 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:42.108505 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 844/1676] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:41 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 314 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:42.111058 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08e432d-479d-49a6-b7f0-bdc23533fe54 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:42.111237 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164f3dcf-d5ec-48f3-8d5c-e1e3fb5a5ccc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:42.111945 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164f3dcf-d5ec-48f3-8d5c-e1e3fb5a5ccc admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:42.112237 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164f3dcf-d5ec-48f3-8d5c-e1e3fb5a5ccc admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:42.112513 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164f3dcf-d5ec-48f3-8d5c-e1e3fb5a5ccc admin admin] PATH_INFO: `/v3/projects/550365774da34580a8a68ff7b2134a7b/users/730bb359b92f4d9499eef69b5a6b30bf/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:42.127003 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08e432d-479d-49a6-b7f0-bdc23533fe54 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:42.134807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b42ddee-7291-440e-92fe-a53fb45c72fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:42.135765 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b42ddee-7291-440e-92fe-a53fb45c72fb admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:42.136570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b42ddee-7291-440e-92fe-a53fb45c72fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:42.137110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b42ddee-7291-440e-92fe-a53fb45c72fb admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:42.138873 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b42ddee-7291-440e-92fe-a53fb45c72fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:42.141715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b42ddee-7291-440e-92fe-a53fb45c72fb admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:42.145031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164f3dcf-d5ec-48f3-8d5c-e1e3fb5a5ccc admin admin] RBAC: Authorizing `identity:create_grant(project_id=550365774da34580a8a68ff7b2134a7b, user_id=730bb359b92f4d9499eef69b5a6b30bf, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:42.147796 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164f3dcf-d5ec-48f3-8d5c-e1e3fb5a5ccc admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:42.182938 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 833/1677] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:42 2022] PUT /identity/v3/projects/550365774da34580a8a68ff7b2134a7b/users/730bb359b92f4d9499eef69b5a6b30bf/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:42.185630 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e00e37-9c30-4a38-96e1-bb05441598f4 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:42.185630 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e00e37-9c30-4a38-96e1-bb05441598f4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:42.185630 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e00e37-9c30-4a38-96e1-bb05441598f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:42.231613 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 845/1678] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:42 2022] POST /identity/v3/projects => generated 361 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:42.244207 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b42ddee-7291-440e-92fe-a53fb45c72fb admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:42.255475 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-62e00e37-9c30-4a38-96e1-bb05441598f4 None None] MFA Rules not processed for user `730bb359b92f4d9499eef69b5a6b30bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:42.257186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-62e00e37-9c30-4a38-96e1-bb05441598f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:42.409849 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:42.431572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 834/1679] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:42 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 248 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:42.432666 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e00e37-9c30-4a38-96e1-bb05441598f4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:42.449524 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b42ddee-7291-440e-92fe-a53fb45c72fb admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:42.450012 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e00e37-9c30-4a38-96e1-bb05441598f4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:42.456997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94dccd9f-50d9-49f3-9b1e-11bfc6d47f43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:42.457635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94dccd9f-50d9-49f3-9b1e-11bfc6d47f43 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:42.457921 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94dccd9f-50d9-49f3-9b1e-11bfc6d47f43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:42.458575 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94dccd9f-50d9-49f3-9b1e-11bfc6d47f43 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:42.459782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94dccd9f-50d9-49f3-9b1e-11bfc6d47f43 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:42.462010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94dccd9f-50d9-49f3-9b1e-11bfc6d47f43 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:42.468862 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff94d69-8f18-48d0-9583-03c7f950df1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:42.469525 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff94d69-8f18-48d0-9583-03c7f950df1b admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:42.469780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff94d69-8f18-48d0-9583-03c7f950df1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:42.470054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff94d69-8f18-48d0-9583-03c7f950df1b admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:42.470694 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff94d69-8f18-48d0-9583-03c7f950df1b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:42.479876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff94d69-8f18-48d0-9583-03c7f950df1b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:42.517466 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 835/1680] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:42 2022] POST /identity/v3/projects => generated 381 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:42.523535 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94dccd9f-50d9-49f3-9b1e-11bfc6d47f43 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:42.572314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 846/1681] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:42 2022] POST /identity/v3/users => generated 287 bytes in 333 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:42.576311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff94d69-8f18-48d0-9583-03c7f950df1b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:42.599062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff94d69-8f18-48d0-9583-03c7f950df1b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:42.618541 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:42.720019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94dccd9f-50d9-49f3-9b1e-11bfc6d47f43 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:42.751979 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8da5d75-798c-4d4a-b93a-23ed68b9f4b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:42.758897 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8da5d75-798c-4d4a-b93a-23ed68b9f4b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:42.759206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8da5d75-798c-4d4a-b93a-23ed68b9f4b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:42.759489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8da5d75-798c-4d4a-b93a-23ed68b9f4b0 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:42.760160 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8da5d75-798c-4d4a-b93a-23ed68b9f4b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:42.777796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928d8861-0837-4bcb-96a0-8d8dbb6f2238 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:42.778487 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928d8861-0837-4bcb-96a0-8d8dbb6f2238 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:42.779548 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928d8861-0837-4bcb-96a0-8d8dbb6f2238 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:42.780476 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928d8861-0837-4bcb-96a0-8d8dbb6f2238 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:42.781214 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928d8861-0837-4bcb-96a0-8d8dbb6f2238 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:42.782589 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8da5d75-798c-4d4a-b93a-23ed68b9f4b0 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:42.783924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928d8861-0837-4bcb-96a0-8d8dbb6f2238 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:42.810408 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 847/1682] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:42 2022] GET /identity/v3/roles => generated 1420 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:18:42.810408 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928d8861-0837-4bcb-96a0-8d8dbb6f2238 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:42.815884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-928d8861-0837-4bcb-96a0-8d8dbb6f2238 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:42.842064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928d8861-0837-4bcb-96a0-8d8dbb6f2238 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:42.852981 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba548fae-6095-4a70-bc46-9fb51da8d16a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:42.855038 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba548fae-6095-4a70-bc46-9fb51da8d16a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:42.855332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba548fae-6095-4a70-bc46-9fb51da8d16a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:42.855611 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba548fae-6095-4a70-bc46-9fb51da8d16a admin admin] PATH_INFO: `/v3/projects/768dedda6202441c89b6d64055c98913` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:42.857915 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba548fae-6095-4a70-bc46-9fb51da8d16a admin admin] RBAC: Authorizing `identity:delete_project(project_id=768dedda6202441c89b6d64055c98913)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:42.863267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba548fae-6095-4a70-bc46-9fb51da8d16a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:42.877392 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 836/1683] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:42 2022] POST /identity/v3/users => generated 297 bytes in 355 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:42.878529 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8da5d75-798c-4d4a-b93a-23ed68b9f4b0 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:42.925492 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8da5d75-798c-4d4a-b93a-23ed68b9f4b0 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:42.933868 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549652fb-eecf-4f2a-8e0a-9c1613ab0707 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:42.934488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549652fb-eecf-4f2a-8e0a-9c1613ab0707 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:42.934774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549652fb-eecf-4f2a-8e0a-9c1613ab0707 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:42.935087 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549652fb-eecf-4f2a-8e0a-9c1613ab0707 admin admin] PATH_INFO: `/v3/projects/bc1f33ab6f9e4319ad62cd516201ecfa/users/b70555c0048f4917bf76f95dec24e581/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:42.964422 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549652fb-eecf-4f2a-8e0a-9c1613ab0707 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc1f33ab6f9e4319ad62cd516201ecfa, user_id=b70555c0048f4917bf76f95dec24e581, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:42.968440 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549652fb-eecf-4f2a-8e0a-9c1613ab0707 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:43.001086 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 837/1684] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:42 2022] PUT /identity/v3/projects/bc1f33ab6f9e4319ad62cd516201ecfa/users/b70555c0048f4917bf76f95dec24e581/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:43.002419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549652fb-eecf-4f2a-8e0a-9c1613ab0707 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:43.019116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ba548fae-6095-4a70-bc46-9fb51da8d16a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '768dedda6202441c89b6d64055c98913', 'request_id': 'req-ba548fae-6095-4a70-bc46-9fb51da8d16a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:43.057329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ba548fae-6095-4a70-bc46-9fb51da8d16a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '768dedda6202441c89b6d64055c98913', 'request_id': 'req-ba548fae-6095-4a70-bc46-9fb51da8d16a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:43.074508 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ba548fae-6095-4a70-bc46-9fb51da8d16a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '768dedda6202441c89b6d64055c98913', 'request_id': 'req-ba548fae-6095-4a70-bc46-9fb51da8d16a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:43.105183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ba548fae-6095-4a70-bc46-9fb51da8d16a admin admin] The token cache is being invalidate because project 768dedda6202441c89b6d64055c98913 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:18:43.106933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ba548fae-6095-4a70-bc46-9fb51da8d16a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:43.108267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ba548fae-6095-4a70-bc46-9fb51da8d16a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:43.111607 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 848/1685] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:18:42 2022] DELETE /identity/v3/projects/768dedda6202441c89b6d64055c98913 => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:43.112548 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba548fae-6095-4a70-bc46-9fb51da8d16a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:43.113916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ba548fae-6095-4a70-bc46-9fb51da8d16a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:43.193726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549652fb-eecf-4f2a-8e0a-9c1613ab0707 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:43.206064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60896ce9-974c-447a-b309-684a58f10cf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:43.206695 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60896ce9-974c-447a-b309-684a58f10cf1 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:43.207001 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60896ce9-974c-447a-b309-684a58f10cf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:43.207590 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60896ce9-974c-447a-b309-684a58f10cf1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:43.210699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60896ce9-974c-447a-b309-684a58f10cf1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:43.214789 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60896ce9-974c-447a-b309-684a58f10cf1 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:43.240759 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 838/1686] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:43 2022] GET /identity/v3/roles => generated 1420 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:43.245704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e91ca8-b51b-4405-8cef-c5cf9a630220 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:43.246003 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e91ca8-b51b-4405-8cef-c5cf9a630220 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:43.246268 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e91ca8-b51b-4405-8cef-c5cf9a630220 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:43.265709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba548fae-6095-4a70-bc46-9fb51da8d16a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:43.269561 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-57e91ca8-b51b-4405-8cef-c5cf9a630220 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:43.270986 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-57e91ca8-b51b-4405-8cef-c5cf9a630220 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:43.278844 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bc3fe0-1f60-4994-8d4b-a5137b5ece8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:43.279764 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bc3fe0-1f60-4994-8d4b-a5137b5ece8a admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:43.280089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bc3fe0-1f60-4994-8d4b-a5137b5ece8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:43.280997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bc3fe0-1f60-4994-8d4b-a5137b5ece8a admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:43.283126 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bc3fe0-1f60-4994-8d4b-a5137b5ece8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:43.291087 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 839/1687] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:43.291087 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e91ca8-b51b-4405-8cef-c5cf9a630220 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:43.291087 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-57e91ca8-b51b-4405-8cef-c5cf9a630220 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:43.294748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bc3fe0-1f60-4994-8d4b-a5137b5ece8a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:43.305388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e91ca8-b51b-4405-8cef-c5cf9a630220 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:43.310674 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9ac0ee-7b7a-4ac3-b0f0-6d9ea5062ee6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:43.313075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9ac0ee-7b7a-4ac3-b0f0-6d9ea5062ee6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:43.313491 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9ac0ee-7b7a-4ac3-b0f0-6d9ea5062ee6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:43.313872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9ac0ee-7b7a-4ac3-b0f0-6d9ea5062ee6 admin admin] PATH_INFO: `/v3/projects/9aaf62e0d3054a13b78ec1377771699f/users/d92ce2186fe443749ff13ad7e90bcb1f/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:43.324764 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 849/1688] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:43 2022] GET /identity/v3/roles => generated 1420 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:43.326239 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45f4a53-27cc-4e51-a85e-1d8d5f328b69 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:43.328418 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45f4a53-27cc-4e51-a85e-1d8d5f328b69 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:43.330444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45f4a53-27cc-4e51-a85e-1d8d5f328b69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:43.337303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9ac0ee-7b7a-4ac3-b0f0-6d9ea5062ee6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9aaf62e0d3054a13b78ec1377771699f, user_id=d92ce2186fe443749ff13ad7e90bcb1f, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:43.340161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9ac0ee-7b7a-4ac3-b0f0-6d9ea5062ee6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:43.373049 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 840/1689] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:43 2022] PUT /identity/v3/projects/9aaf62e0d3054a13b78ec1377771699f/users/d92ce2186fe443749ff13ad7e90bcb1f/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:43.378450 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9ac0ee-7b7a-4ac3-b0f0-6d9ea5062ee6 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:43.390273 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b45f4a53-27cc-4e51-a85e-1d8d5f328b69 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:43.392546 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b45f4a53-27cc-4e51-a85e-1d8d5f328b69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:43.398984 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:43.442105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:43.506663 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9ac0ee-7b7a-4ac3-b0f0-6d9ea5062ee6 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:43.523329 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4497692d-cc34-469d-9ee3-6fd559912d9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:43.524360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4497692d-cc34-469d-9ee3-6fd559912d9b admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:43.524785 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4497692d-cc34-469d-9ee3-6fd559912d9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:43.525073 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4497692d-cc34-469d-9ee3-6fd559912d9b admin admin] PATH_INFO: `/v3/projects/bc1f33ab6f9e4319ad62cd516201ecfa/users/b70555c0048f4917bf76f95dec24e581/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:43.545869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4497692d-cc34-469d-9ee3-6fd559912d9b admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc1f33ab6f9e4319ad62cd516201ecfa, user_id=b70555c0048f4917bf76f95dec24e581, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:43.548912 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4497692d-cc34-469d-9ee3-6fd559912d9b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:43.553006 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 850/1690] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:43.553940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45f4a53-27cc-4e51-a85e-1d8d5f328b69 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:43.577900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 841/1691] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:43 2022] PUT /identity/v3/projects/bc1f33ab6f9e4319ad62cd516201ecfa/users/b70555c0048f4917bf76f95dec24e581/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:43.579556 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4497692d-cc34-469d-9ee3-6fd559912d9b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:43.584731 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45f4a53-27cc-4e51-a85e-1d8d5f328b69 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:43.788117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb6c46a-6c5f-41d0-bb9b-92e55d02b650 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:43.788879 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb6c46a-6c5f-41d0-bb9b-92e55d02b650 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:43.789802 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb6c46a-6c5f-41d0-bb9b-92e55d02b650 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:43.789802 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb6c46a-6c5f-41d0-bb9b-92e55d02b650 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:43.791807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb6c46a-6c5f-41d0-bb9b-92e55d02b650 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:43.797241 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4497692d-cc34-469d-9ee3-6fd559912d9b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:43.797763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb6c46a-6c5f-41d0-bb9b-92e55d02b650 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:43.812995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e730cb4-e7f1-4f94-934e-deb2e8fd75fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:43.813785 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e730cb4-e7f1-4f94-934e-deb2e8fd75fd admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:43.814669 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e730cb4-e7f1-4f94-934e-deb2e8fd75fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:43.814669 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e730cb4-e7f1-4f94-934e-deb2e8fd75fd admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:43.816517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e730cb4-e7f1-4f94-934e-deb2e8fd75fd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:18:43.816975 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e730cb4-e7f1-4f94-934e-deb2e8fd75fd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:43.818811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e730cb4-e7f1-4f94-934e-deb2e8fd75fd admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:43.836251 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 851/1692] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:43 2022] GET /identity/v3/roles => generated 1420 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:43.838222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34101621-d705-40f8-91eb-1efdabba2c8a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:43.838222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34101621-d705-40f8-91eb-1efdabba2c8a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:43.838538 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34101621-d705-40f8-91eb-1efdabba2c8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:43.862336 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 842/1693] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:18:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:18:43.863235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e730cb4-e7f1-4f94-934e-deb2e8fd75fd admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:43.880781 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e730cb4-e7f1-4f94-934e-deb2e8fd75fd admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:43.896660 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c8f5f4-315a-4637-a312-35d2aab1e8a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:43.897355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c8f5f4-315a-4637-a312-35d2aab1e8a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:43.897640 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c8f5f4-315a-4637-a312-35d2aab1e8a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:43.897952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c8f5f4-315a-4637-a312-35d2aab1e8a1 admin admin] PATH_INFO: `/v3/projects/9aaf62e0d3054a13b78ec1377771699f/users/d92ce2186fe443749ff13ad7e90bcb1f/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:43.899981 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c8f5f4-315a-4637-a312-35d2aab1e8a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9aaf62e0d3054a13b78ec1377771699f, user_id=d92ce2186fe443749ff13ad7e90bcb1f, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:43.903278 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c8f5f4-315a-4637-a312-35d2aab1e8a1 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:43.918997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 843/1694] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:43 2022] PUT /identity/v3/projects/9aaf62e0d3054a13b78ec1377771699f/users/d92ce2186fe443749ff13ad7e90bcb1f/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:43.921401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c8f5f4-315a-4637-a312-35d2aab1e8a1 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:43.925334 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-34101621-d705-40f8-91eb-1efdabba2c8a None None] MFA Rules not processed for user `b70555c0048f4917bf76f95dec24e581`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:43.927090 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-34101621-d705-40f8-91eb-1efdabba2c8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:44.100646 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c8f5f4-315a-4637-a312-35d2aab1e8a1 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:44.112180 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fb883b-6366-455a-bca9-9f78a7473b90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:44.113065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fb883b-6366-455a-bca9-9f78a7473b90 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:44.113644 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fb883b-6366-455a-bca9-9f78a7473b90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:44.114089 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fb883b-6366-455a-bca9-9f78a7473b90 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:44.115094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fb883b-6366-455a-bca9-9f78a7473b90 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:44.117978 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 852/1695] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:43 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:44.121140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f2e2ee-af49-40a3-b018-79bc1e91a6be None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:44.121431 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f2e2ee-af49-40a3-b018-79bc1e91a6be None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:44.121704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f2e2ee-af49-40a3-b018-79bc1e91a6be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:44.132998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fb883b-6366-455a-bca9-9f78a7473b90 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:44.203863 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a6f2e2ee-af49-40a3-b018-79bc1e91a6be None None] MFA Rules not processed for user `d92ce2186fe443749ff13ad7e90bcb1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:44.205493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a6f2e2ee-af49-40a3-b018-79bc1e91a6be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:44.205610 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 844/1696] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:43 2022] POST /identity/v3/projects => generated 377 bytes in 286 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:44.209760 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fb883b-6366-455a-bca9-9f78a7473b90 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:44.215642 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b7fb883b-6366-455a-bca9-9f78a7473b90 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:44.231471 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:44.293525 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:44.351553 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fb883b-6366-455a-bca9-9f78a7473b90 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:44.361100 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b85f37-a0e0-4667-b3eb-a0a8eca4bf15 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:44.361739 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b85f37-a0e0-4667-b3eb-a0a8eca4bf15 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:44.362017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b85f37-a0e0-4667-b3eb-a0a8eca4bf15 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:44.362286 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b85f37-a0e0-4667-b3eb-a0a8eca4bf15 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:44.415760 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 853/1697] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:44 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 297 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:44.419907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f2e2ee-af49-40a3-b018-79bc1e91a6be None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:44.584479 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b85f37-a0e0-4667-b3eb-a0a8eca4bf15 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:44.594388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b85f37-a0e0-4667-b3eb-a0a8eca4bf15 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:44.630285 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f2e2ee-af49-40a3-b018-79bc1e91a6be None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:44.638237 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 845/1698] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:44 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:44.641624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b85f37-a0e0-4667-b3eb-a0a8eca4bf15 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:44.643913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bcc56f-f3e3-41bd-81c6-3a68a009fc47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:44.646076 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-64b85f37-a0e0-4667-b3eb-a0a8eca4bf15 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:44.646752 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bcc56f-f3e3-41bd-81c6-3a68a009fc47 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:44.647147 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bcc56f-f3e3-41bd-81c6-3a68a009fc47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:44.647566 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bcc56f-f3e3-41bd-81c6-3a68a009fc47 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:44.649384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34bcc56f-f3e3-41bd-81c6-3a68a009fc47 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:44.653167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34bcc56f-f3e3-41bd-81c6-3a68a009fc47 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:44.674705 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b85f37-a0e0-4667-b3eb-a0a8eca4bf15 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:44.685405 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdad7f69-7661-4348-9252-5967a5fb75cb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:44.688502 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdad7f69-7661-4348-9252-5967a5fb75cb service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:44.688879 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdad7f69-7661-4348-9252-5967a5fb75cb service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:44.690485 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdad7f69-7661-4348-9252-5967a5fb75cb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:44.712664 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 854/1699] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:44 2022] POST /identity/v3/users => generated 289 bytes in 293 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:44.716809 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bcc56f-f3e3-41bd-81c6-3a68a009fc47 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:44.727036 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdad7f69-7661-4348-9252-5967a5fb75cb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:44.743355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdad7f69-7661-4348-9252-5967a5fb75cb service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:44.782193 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bcc56f-f3e3-41bd-81c6-3a68a009fc47 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:44.791852 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 846/1700] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:44 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:44.801328 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b949293f-1f08-4c7a-b552-80614c7b8a79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:44.802073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b949293f-1f08-4c7a-b552-80614c7b8a79 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:44.802458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b949293f-1f08-4c7a-b552-80614c7b8a79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:44.802837 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b949293f-1f08-4c7a-b552-80614c7b8a79 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:44.803626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b949293f-1f08-4c7a-b552-80614c7b8a79 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:44.817219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b949293f-1f08-4c7a-b552-80614c7b8a79 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:44.845524 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 855/1701] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:44 2022] GET /identity/v3/roles => generated 1420 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:18:44.849285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdad7f69-7661-4348-9252-5967a5fb75cb service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:44.872108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdad7f69-7661-4348-9252-5967a5fb75cb service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:44.897986 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84f30a9-2739-4be9-bf80-00a3fa6d3dc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:44.898622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84f30a9-2739-4be9-bf80-00a3fa6d3dc9 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:44.898927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84f30a9-2739-4be9-bf80-00a3fa6d3dc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:44.899209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84f30a9-2739-4be9-bf80-00a3fa6d3dc9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:44.899861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84f30a9-2739-4be9-bf80-00a3fa6d3dc9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:44.901492 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84f30a9-2739-4be9-bf80-00a3fa6d3dc9 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:44.942400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b949293f-1f08-4c7a-b552-80614c7b8a79 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:44.948665 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 847/1702] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:44 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:18:44.957988 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b949293f-1f08-4c7a-b552-80614c7b8a79 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:44.973121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd825286-736f-4e30-947a-a28005bf4f98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:44.974070 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd825286-736f-4e30-947a-a28005bf4f98 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:44.974781 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd825286-736f-4e30-947a-a28005bf4f98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:44.976314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd825286-736f-4e30-947a-a28005bf4f98 admin admin] PATH_INFO: `/v3/projects/c74ba6325b1e4b49bb9531d3aac902ea/users/d062f99445b64b8a8451ee9e61e68d05/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:45.022795 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd825286-736f-4e30-947a-a28005bf4f98 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c74ba6325b1e4b49bb9531d3aac902ea, user_id=d062f99445b64b8a8451ee9e61e68d05, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:45.026675 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd825286-736f-4e30-947a-a28005bf4f98 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:45.053529 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84f30a9-2739-4be9-bf80-00a3fa6d3dc9 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:45.067979 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84f30a9-2739-4be9-bf80-00a3fa6d3dc9 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:45.069755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 856/1703] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:44 2022] PUT /identity/v3/projects/c74ba6325b1e4b49bb9531d3aac902ea/users/d062f99445b64b8a8451ee9e61e68d05/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:45.082002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba8a1c4-875b-41ee-9932-e1859ffc0007 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:45.082383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba8a1c4-875b-41ee-9932-e1859ffc0007 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:45.083141 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba8a1c4-875b-41ee-9932-e1859ffc0007 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:45.195223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-eba8a1c4-875b-41ee-9932-e1859ffc0007 None None] MFA Rules not processed for user `d062f99445b64b8a8451ee9e61e68d05`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:45.196505 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-eba8a1c4-875b-41ee-9932-e1859ffc0007 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:45.251252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86e0359-a96d-4dd9-aaab-3c02599c3e07 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:45.251992 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86e0359-a96d-4dd9-aaab-3c02599c3e07 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:45.252277 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86e0359-a96d-4dd9-aaab-3c02599c3e07 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:45.252550 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86e0359-a96d-4dd9-aaab-3c02599c3e07 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:45.261485 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e86e0359-a96d-4dd9-aaab-3c02599c3e07 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:45.386384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 857/1704] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:45 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 305 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:18:45.391136 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba8a1c4-875b-41ee-9932-e1859ffc0007 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:45.402145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:45.447333 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:45.460780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86e0359-a96d-4dd9-aaab-3c02599c3e07 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:45.462262 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86e0359-a96d-4dd9-aaab-3c02599c3e07 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:45.507798 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 848/1705] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:45 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 456 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:18:45.586212 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba8a1c4-875b-41ee-9932-e1859ffc0007 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:45.604685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1671b4-f7a1-44fb-840b-1141d03846de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:45.606251 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1671b4-f7a1-44fb-840b-1141d03846de admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:45.606251 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1671b4-f7a1-44fb-840b-1141d03846de admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:45.606251 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1671b4-f7a1-44fb-840b-1141d03846de admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:45.611317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1671b4-f7a1-44fb-840b-1141d03846de admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:45.613733 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1671b4-f7a1-44fb-840b-1141d03846de admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:45.704629 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 858/1706] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:45 2022] POST /identity/v3/projects => generated 375 bytes in 314 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:45.705819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86e0359-a96d-4dd9-aaab-3c02599c3e07 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:45.735118 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86e0359-a96d-4dd9-aaab-3c02599c3e07 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:45.939494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1671b4-f7a1-44fb-840b-1141d03846de admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:45.941304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-cc1671b4-f7a1-44fb-840b-1141d03846de admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:46.018397 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63feaa12-c7b1-4ab5-9199-ec49bcca3810 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:46.023415 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63feaa12-c7b1-4ab5-9199-ec49bcca3810 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:46.023871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63feaa12-c7b1-4ab5-9199-ec49bcca3810 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:46.025911 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63feaa12-c7b1-4ab5-9199-ec49bcca3810 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:46.030090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63feaa12-c7b1-4ab5-9199-ec49bcca3810 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:46.037202 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63feaa12-c7b1-4ab5-9199-ec49bcca3810 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:46.113686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1671b4-f7a1-44fb-840b-1141d03846de admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:46.121094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 849/1707] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:45 2022] POST /identity/v3/users => generated 294 bytes in 416 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:46.122643 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63feaa12-c7b1-4ab5-9199-ec49bcca3810 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:46.128321 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631dac2c-eb7c-4bcf-878d-20c49a30602a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:46.130002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631dac2c-eb7c-4bcf-878d-20c49a30602a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:46.130295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631dac2c-eb7c-4bcf-878d-20c49a30602a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:46.131729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631dac2c-eb7c-4bcf-878d-20c49a30602a admin admin] PATH_INFO: `/v3/users/21815172a4ea4eca8cf5b67415e3b1ff` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:46.138416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631dac2c-eb7c-4bcf-878d-20c49a30602a admin admin] RBAC: Authorizing `identity:delete_user(user_id=21815172a4ea4eca8cf5b67415e3b1ff)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:46.145176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631dac2c-eb7c-4bcf-878d-20c49a30602a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:46.180006 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63feaa12-c7b1-4ab5-9199-ec49bcca3810 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:46.200338 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39daf824-6fd5-4c43-b2ac-e4a567cadc89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:46.201010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39daf824-6fd5-4c43-b2ac-e4a567cadc89 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:46.201436 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39daf824-6fd5-4c43-b2ac-e4a567cadc89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:46.201717 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39daf824-6fd5-4c43-b2ac-e4a567cadc89 admin admin] PATH_INFO: `/v3/users/31ce4daf5ef64741ba5b4dd363453096` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:46.204102 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39daf824-6fd5-4c43-b2ac-e4a567cadc89 admin admin] RBAC: Authorizing `identity:delete_user(user_id=31ce4daf5ef64741ba5b4dd363453096)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:46.211704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39daf824-6fd5-4c43-b2ac-e4a567cadc89 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:46.266793 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-631dac2c-eb7c-4bcf-878d-20c49a30602a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21815172a4ea4eca8cf5b67415e3b1ff'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:46.268162 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-631dac2c-eb7c-4bcf-878d-20c49a30602a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21815172a4ea4eca8cf5b67415e3b1ff'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:46.293954 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-631dac2c-eb7c-4bcf-878d-20c49a30602a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21815172a4ea4eca8cf5b67415e3b1ff'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:46.295055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-631dac2c-eb7c-4bcf-878d-20c49a30602a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21815172a4ea4eca8cf5b67415e3b1ff'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:46.322000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-631dac2c-eb7c-4bcf-878d-20c49a30602a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21815172a4ea4eca8cf5b67415e3b1ff'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:46.358530 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-39daf824-6fd5-4c43-b2ac-e4a567cadc89 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '31ce4daf5ef64741ba5b4dd363453096'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:46.361793 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 859/1708] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:18:45 2022] DELETE /identity/v3/users/21815172a4ea4eca8cf5b67415e3b1ff => generated 0 bytes in 419 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:46.362033 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-39daf824-6fd5-4c43-b2ac-e4a567cadc89 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '31ce4daf5ef64741ba5b4dd363453096'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:46.362749 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631dac2c-eb7c-4bcf-878d-20c49a30602a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:46.369143 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-631dac2c-eb7c-4bcf-878d-20c49a30602a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:46.372708 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-39daf824-6fd5-4c43-b2ac-e4a567cadc89 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '31ce4daf5ef64741ba5b4dd363453096'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:46.373380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-39daf824-6fd5-4c43-b2ac-e4a567cadc89 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '31ce4daf5ef64741ba5b4dd363453096'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:46.388112 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-39daf824-6fd5-4c43-b2ac-e4a567cadc89 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '31ce4daf5ef64741ba5b4dd363453096'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:46.430041 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:46.438767 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 850/1709] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:18:46 2022] DELETE /identity/v3/users/31ce4daf5ef64741ba5b4dd363453096 => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:46.445468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a506a2-6ec5-4612-8c79-4e4fccd25718 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:46.445468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a506a2-6ec5-4612-8c79-4e4fccd25718 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:46.445717 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a506a2-6ec5-4612-8c79-4e4fccd25718 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:46.489640 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-71a506a2-6ec5-4612-8c79-4e4fccd25718 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:46.490936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-71a506a2-6ec5-4612-8c79-4e4fccd25718 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:46.553257 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631dac2c-eb7c-4bcf-878d-20c49a30602a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:46.620153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:46.654252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 851/1710] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:46.655402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f2eb74-8d5d-48c4-89e0-3f4b5eeee737 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:46.655686 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f2eb74-8d5d-48c4-89e0-3f4b5eeee737 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:46.656556 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f2eb74-8d5d-48c4-89e0-3f4b5eeee737 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:46.694070 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ad3bf3-51eb-4370-a778-1a70e13fab82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:46.694875 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ad3bf3-51eb-4370-a778-1a70e13fab82 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:46.695162 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ad3bf3-51eb-4370-a778-1a70e13fab82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:46.695438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ad3bf3-51eb-4370-a778-1a70e13fab82 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:46.696100 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ad3bf3-51eb-4370-a778-1a70e13fab82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:46.697728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ad3bf3-51eb-4370-a778-1a70e13fab82 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:46.703886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c8f2eb74-8d5d-48c4-89e0-3f4b5eeee737 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:46.705139 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c8f2eb74-8d5d-48c4-89e0-3f4b5eeee737 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:46.720706 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 860/1711] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:46 2022] GET /identity/v3/roles => generated 1420 bytes in 358 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:46.724786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ad3bf3-51eb-4370-a778-1a70e13fab82 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:46.729204 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 852/1712] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:46.730125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f2eb74-8d5d-48c4-89e0-3f4b5eeee737 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:46.731552 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c8f2eb74-8d5d-48c4-89e0-3f4b5eeee737 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:46.736519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c4ad3bf3-51eb-4370-a778-1a70e13fab82 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:46.782165 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f2eb74-8d5d-48c4-89e0-3f4b5eeee737 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:46.793369 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2364d68-acaf-44b5-8127-f4b98b055d9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:46.794072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2364d68-acaf-44b5-8127-f4b98b055d9b admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:46.794413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2364d68-acaf-44b5-8127-f4b98b055d9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:46.794688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2364d68-acaf-44b5-8127-f4b98b055d9b admin admin] PATH_INFO: `/v3/projects/e19cea5ab3524c1091f8c3e10f812086/users/99327d68a5f845b0a44224911ecb7f3b/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:46.822558 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2364d68-acaf-44b5-8127-f4b98b055d9b admin admin] RBAC: Authorizing `identity:create_grant(project_id=e19cea5ab3524c1091f8c3e10f812086, user_id=99327d68a5f845b0a44224911ecb7f3b, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:46.825290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2364d68-acaf-44b5-8127-f4b98b055d9b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:46.857063 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 853/1713] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:46 2022] PUT /identity/v3/projects/e19cea5ab3524c1091f8c3e10f812086/users/99327d68a5f845b0a44224911ecb7f3b/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:46.857063 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2364d68-acaf-44b5-8127-f4b98b055d9b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:46.858519 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e2364d68-acaf-44b5-8127-f4b98b055d9b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:46.884046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ad3bf3-51eb-4370-a778-1a70e13fab82 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:46.991966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c79dea-9396-4fd7-8a98-6621e166137d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:46.992595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c79dea-9396-4fd7-8a98-6621e166137d service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:46.992879 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c79dea-9396-4fd7-8a98-6621e166137d service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:46.993154 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c79dea-9396-4fd7-8a98-6621e166137d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:46.996138 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2364d68-acaf-44b5-8127-f4b98b055d9b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:47.006470 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1834de3-dcba-47a7-a63c-ee1467704ef8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:47.007185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1834de3-dcba-47a7-a63c-ee1467704ef8 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:47.007423 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1834de3-dcba-47a7-a63c-ee1467704ef8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:47.007693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1834de3-dcba-47a7-a63c-ee1467704ef8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:47.062705 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:47.135403 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c79dea-9396-4fd7-8a98-6621e166137d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:47.136592 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c79dea-9396-4fd7-8a98-6621e166137d service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:47.153142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1834de3-dcba-47a7-a63c-ee1467704ef8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:47.154597 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1834de3-dcba-47a7-a63c-ee1467704ef8 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:47.156785 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 861/1714] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:47.157653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c79dea-9396-4fd7-8a98-6621e166137d service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:47.167467 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 854/1715] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:47.178112 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c79dea-9396-4fd7-8a98-6621e166137d service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:47.196767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fabe34-911b-4756-91dc-5704ce3880cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:47.197461 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fabe34-911b-4756-91dc-5704ce3880cf admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:47.197745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fabe34-911b-4756-91dc-5704ce3880cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:47.198023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fabe34-911b-4756-91dc-5704ce3880cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:47.198678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fabe34-911b-4756-91dc-5704ce3880cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:47.200580 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fabe34-911b-4756-91dc-5704ce3880cf admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:47.223807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 862/1716] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:47 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:47.225263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1834de3-dcba-47a7-a63c-ee1467704ef8 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:47.239629 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1834de3-dcba-47a7-a63c-ee1467704ef8 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:47.251465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4533cf11-9e5f-4cc6-a8c7-05a7956e74ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:47.252162 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4533cf11-9e5f-4cc6-a8c7-05a7956e74ca admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:47.252464 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4533cf11-9e5f-4cc6-a8c7-05a7956e74ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:47.252706 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4533cf11-9e5f-4cc6-a8c7-05a7956e74ca admin admin] PATH_INFO: `/v3/projects/e19cea5ab3524c1091f8c3e10f812086/users/99327d68a5f845b0a44224911ecb7f3b/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:47.255417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4533cf11-9e5f-4cc6-a8c7-05a7956e74ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=e19cea5ab3524c1091f8c3e10f812086, user_id=99327d68a5f845b0a44224911ecb7f3b, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:47.259419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4533cf11-9e5f-4cc6-a8c7-05a7956e74ca admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:47.279559 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 855/1717] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:47 2022] PUT /identity/v3/projects/e19cea5ab3524c1091f8c3e10f812086/users/99327d68a5f845b0a44224911ecb7f3b/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:47.290164 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde88bc1-a254-407c-92fc-414cf211e7de None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:47.290451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde88bc1-a254-407c-92fc-414cf211e7de None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:47.290728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde88bc1-a254-407c-92fc-414cf211e7de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:47.356711 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-cde88bc1-a254-407c-92fc-414cf211e7de None None] MFA Rules not processed for user `99327d68a5f845b0a44224911ecb7f3b`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:47.363253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-cde88bc1-a254-407c-92fc-414cf211e7de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:47.553579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 863/1718] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:47 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 265 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:47.579661 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4533cf11-9e5f-4cc6-a8c7-05a7956e74ca admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:47.579661 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4533cf11-9e5f-4cc6-a8c7-05a7956e74ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:47.714311 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4533cf11-9e5f-4cc6-a8c7-05a7956e74ca admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:47.719912 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef43b8b-6bc2-4c79-a774-6ecbdc071392 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:47.720867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef43b8b-6bc2-4c79-a774-6ecbdc071392 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:47.721316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef43b8b-6bc2-4c79-a774-6ecbdc071392 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:47.722819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef43b8b-6bc2-4c79-a774-6ecbdc071392 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:47.726157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0ef43b8b-6bc2-4c79-a774-6ecbdc071392 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:47.744971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:47.917407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef43b8b-6bc2-4c79-a774-6ecbdc071392 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:47.919648 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef43b8b-6bc2-4c79-a774-6ecbdc071392 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:47.959233 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 856/1719] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:47 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:48.357623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bbaafe-2fe4-4ea9-a4fe-3607a3f705e2 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:48.358577 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bbaafe-2fe4-4ea9-a4fe-3607a3f705e2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:48.359110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bbaafe-2fe4-4ea9-a4fe-3607a3f705e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:48.421801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-32bbaafe-2fe4-4ea9-a4fe-3607a3f705e2 None None] MFA Rules not processed for user `349e19300f1e48c3ac44fc34611c7f4a`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:48.423123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-32bbaafe-2fe4-4ea9-a4fe-3607a3f705e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:48.549288 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ede4545-d9db-4e5c-a429-ddebdbcdd4f1 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:48.552808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ede4545-d9db-4e5c-a429-ddebdbcdd4f1 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:48.553132 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ede4545-d9db-4e5c-a429-ddebdbcdd4f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:48.665521 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-2ede4545-d9db-4e5c-a429-ddebdbcdd4f1 None None] MFA Rules not processed for user `0c5dcfa5794843a5896fa051a4554e1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:48.666098 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 864/1720] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:48 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 310 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:18:48.674846 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2ede4545-d9db-4e5c-a429-ddebdbcdd4f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:48.903448 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 857/1721] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:48 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 346 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:48.905112 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b6ed2f-1983-420f-9b09-6be925933226 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:48.905357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b6ed2f-1983-420f-9b09-6be925933226 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:48.905643 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b6ed2f-1983-420f-9b09-6be925933226 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:48.961847 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c4b6ed2f-1983-420f-9b09-6be925933226 None None] MFA Rules not processed for user `0c5dcfa5794843a5896fa051a4554e1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:48.969153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ede4545-d9db-4e5c-a429-ddebdbcdd4f1 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:48.970635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2ede4545-d9db-4e5c-a429-ddebdbcdd4f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:48.984755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c4b6ed2f-1983-420f-9b09-6be925933226 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:49.025169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 865/1722] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:48 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:49.045642 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b6ed2f-1983-420f-9b09-6be925933226 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:49.047897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c4b6ed2f-1983-420f-9b09-6be925933226 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:49.141011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ede4545-d9db-4e5c-a429-ddebdbcdd4f1 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:49.155215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd7862b-bd8d-461f-9c12-731d98f534da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:49.156412 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd7862b-bd8d-461f-9c12-731d98f534da admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:49.158146 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd7862b-bd8d-461f-9c12-731d98f534da admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:49.159834 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd7862b-bd8d-461f-9c12-731d98f534da admin admin] PATH_INFO: `/v3/users/730bb359b92f4d9499eef69b5a6b30bf` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:49.160059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:49.162390 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd7862b-bd8d-461f-9c12-731d98f534da admin admin] RBAC: Authorizing `identity:delete_user(user_id=730bb359b92f4d9499eef69b5a6b30bf)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:49.164447 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd7862b-bd8d-461f-9c12-731d98f534da admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:49.267259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b6ed2f-1983-420f-9b09-6be925933226 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:49.296600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e574124-f781-4f79-bf18-58b052915110 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:49.300826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e574124-f781-4f79-bf18-58b052915110 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:49.301534 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e574124-f781-4f79-bf18-58b052915110 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:49.302144 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e574124-f781-4f79-bf18-58b052915110 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:49.334226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:49.338402 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e574124-f781-4f79-bf18-58b052915110 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:49.347111 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bfd7862b-bd8d-461f-9c12-731d98f534da admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '730bb359b92f4d9499eef69b5a6b30bf'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:49.352159 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e574124-f781-4f79-bf18-58b052915110 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:49.352850 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bfd7862b-bd8d-461f-9c12-731d98f534da admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '730bb359b92f4d9499eef69b5a6b30bf'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:49.378757 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 866/1723] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:49 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:18:49.380245 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bfd7862b-bd8d-461f-9c12-731d98f534da admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '730bb359b92f4d9499eef69b5a6b30bf'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:49.381005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bfd7862b-bd8d-461f-9c12-731d98f534da admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '730bb359b92f4d9499eef69b5a6b30bf'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:49.385421 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e574124-f781-4f79-bf18-58b052915110 service nova] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:49.392266 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8e574124-f781-4f79-bf18-58b052915110 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:49.398434 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bfd7862b-bd8d-461f-9c12-731d98f534da admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '730bb359b92f4d9499eef69b5a6b30bf'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:49.429222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e574124-f781-4f79-bf18-58b052915110 service nova] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:49.443242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174261cb-dc4a-4a09-a2d1-7639f6b38baa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:49.444000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174261cb-dc4a-4a09-a2d1-7639f6b38baa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:49.444488 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174261cb-dc4a-4a09-a2d1-7639f6b38baa admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:49.444832 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174261cb-dc4a-4a09-a2d1-7639f6b38baa admin admin] PATH_INFO: `/v3/projects/2031a924ca5c4eb9919e291aea2178bb` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:49.448906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 858/1724] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:18:48 2022] DELETE /identity/v3/users/730bb359b92f4d9499eef69b5a6b30bf => generated 0 bytes in 480 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:18:49.450024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174261cb-dc4a-4a09-a2d1-7639f6b38baa admin admin] RBAC: Authorizing `identity:delete_project(project_id=2031a924ca5c4eb9919e291aea2178bb)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:49.453996 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b9c046-f169-4014-86f6-bb266085ab2d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:49.454325 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b9c046-f169-4014-86f6-bb266085ab2d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:49.456242 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b9c046-f169-4014-86f6-bb266085ab2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:49.457951 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174261cb-dc4a-4a09-a2d1-7639f6b38baa admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:49.505426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-08b9c046-f169-4014-86f6-bb266085ab2d None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:49.509009 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-08b9c046-f169-4014-86f6-bb266085ab2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:49.587400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-174261cb-dc4a-4a09-a2d1-7639f6b38baa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2031a924ca5c4eb9919e291aea2178bb', 'request_id': 'req-174261cb-dc4a-4a09-a2d1-7639f6b38baa'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:49.601348 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-174261cb-dc4a-4a09-a2d1-7639f6b38baa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2031a924ca5c4eb9919e291aea2178bb', 'request_id': 'req-174261cb-dc4a-4a09-a2d1-7639f6b38baa'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:49.616745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-174261cb-dc4a-4a09-a2d1-7639f6b38baa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2031a924ca5c4eb9919e291aea2178bb', 'request_id': 'req-174261cb-dc4a-4a09-a2d1-7639f6b38baa'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:49.651981 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-174261cb-dc4a-4a09-a2d1-7639f6b38baa admin admin] The token cache is being invalidate because project 2031a924ca5c4eb9919e291aea2178bb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:18:49.653998 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-174261cb-dc4a-4a09-a2d1-7639f6b38baa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:49.656360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-174261cb-dc4a-4a09-a2d1-7639f6b38baa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:49.658465 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 867/1725] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:18:49 2022] DELETE /identity/v3/projects/2031a924ca5c4eb9919e291aea2178bb => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:49.664632 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174261cb-dc4a-4a09-a2d1-7639f6b38baa admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:49.675311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-174261cb-dc4a-4a09-a2d1-7639f6b38baa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:49.706353 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 859/1726] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:49.754265 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b9c046-f169-4014-86f6-bb266085ab2d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:49.755735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-08b9c046-f169-4014-86f6-bb266085ab2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:49.889280 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174261cb-dc4a-4a09-a2d1-7639f6b38baa admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:49.903091 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b9c046-f169-4014-86f6-bb266085ab2d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:49.916788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3238c6c7-0cc4-4d69-b773-989e84071795 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:49.917704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3238c6c7-0cc4-4d69-b773-989e84071795 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:49.918131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3238c6c7-0cc4-4d69-b773-989e84071795 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:49.918574 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3238c6c7-0cc4-4d69-b773-989e84071795 admin admin] PATH_INFO: `/v3/users/b70555c0048f4917bf76f95dec24e581` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:49.920009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3238c6c7-0cc4-4d69-b773-989e84071795 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b70555c0048f4917bf76f95dec24e581)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:49.922207 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3238c6c7-0cc4-4d69-b773-989e84071795 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:50.042962 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3238c6c7-0cc4-4d69-b773-989e84071795 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b70555c0048f4917bf76f95dec24e581'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:50.045006 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3238c6c7-0cc4-4d69-b773-989e84071795 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b70555c0048f4917bf76f95dec24e581'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:50.049497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:50.071080 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3238c6c7-0cc4-4d69-b773-989e84071795 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b70555c0048f4917bf76f95dec24e581'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:50.071792 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3238c6c7-0cc4-4d69-b773-989e84071795 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b70555c0048f4917bf76f95dec24e581'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:50.115815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3238c6c7-0cc4-4d69-b773-989e84071795 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b70555c0048f4917bf76f95dec24e581'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:50.141280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa710b70-3465-41a3-a868-591479698a7e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:50.141918 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa710b70-3465-41a3-a868-591479698a7e service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:50.142200 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa710b70-3465-41a3-a868-591479698a7e service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:50.142581 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa710b70-3465-41a3-a868-591479698a7e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:50.146503 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fa710b70-3465-41a3-a868-591479698a7e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:50.166069 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 868/1727] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:18:49 2022] DELETE /identity/v3/users/b70555c0048f4917bf76f95dec24e581 => generated 0 bytes in 500 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:50.172746 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3238c6c7-0cc4-4d69-b773-989e84071795 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:50.175020 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3238c6c7-0cc4-4d69-b773-989e84071795 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:50.184235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa710b70-3465-41a3-a868-591479698a7e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:50.185747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa710b70-3465-41a3-a868-591479698a7e service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:50.207658 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 860/1728] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:50.339749 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3238c6c7-0cc4-4d69-b773-989e84071795 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:50.350834 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8a3d03-cd3b-423f-a76d-5a4b03a2c27d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:50.351683 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8a3d03-cd3b-423f-a76d-5a4b03a2c27d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:50.351970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8a3d03-cd3b-423f-a76d-5a4b03a2c27d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:50.352250 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8a3d03-cd3b-423f-a76d-5a4b03a2c27d admin admin] PATH_INFO: `/v3/projects/d09aabd4960043f5ae0efae870db28fb` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:50.355056 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8a3d03-cd3b-423f-a76d-5a4b03a2c27d admin admin] RBAC: Authorizing `identity:delete_project(project_id=d09aabd4960043f5ae0efae870db28fb)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:50.358737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8a3d03-cd3b-423f-a76d-5a4b03a2c27d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:50.372430 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fcff12-a124-4ff0-b3cb-2d09e9f0ac1b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:50.373016 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fcff12-a124-4ff0-b3cb-2d09e9f0ac1b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:50.373435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fcff12-a124-4ff0-b3cb-2d09e9f0ac1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:50.437874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-75fcff12-a124-4ff0-b3cb-2d09e9f0ac1b None None] MFA Rules not processed for user `e2846ad486314958bb74af71b647c470`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:50.440419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-75fcff12-a124-4ff0-b3cb-2d09e9f0ac1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:50.446193 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0b8a3d03-cd3b-423f-a76d-5a4b03a2c27d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd09aabd4960043f5ae0efae870db28fb', 'request_id': 'req-0b8a3d03-cd3b-423f-a76d-5a4b03a2c27d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:50.455952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:DELETE=4,keystone:SELECT=95,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:50.463065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0b8a3d03-cd3b-423f-a76d-5a4b03a2c27d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd09aabd4960043f5ae0efae870db28fb', 'request_id': 'req-0b8a3d03-cd3b-423f-a76d-5a4b03a2c27d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:50.476859 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0b8a3d03-cd3b-423f-a76d-5a4b03a2c27d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd09aabd4960043f5ae0efae870db28fb', 'request_id': 'req-0b8a3d03-cd3b-423f-a76d-5a4b03a2c27d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:50.504719 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0b8a3d03-cd3b-423f-a76d-5a4b03a2c27d admin admin] The token cache is being invalidate because project d09aabd4960043f5ae0efae870db28fb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:18:50.505472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0b8a3d03-cd3b-423f-a76d-5a4b03a2c27d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:50.505911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0b8a3d03-cd3b-423f-a76d-5a4b03a2c27d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:50.508918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 869/1729] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:18:50 2022] DELETE /identity/v3/projects/d09aabd4960043f5ae0efae870db28fb => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:50.513841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1235453-2e8a-4618-a438-709d33b4c625 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:50.514118 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1235453-2e8a-4618-a438-709d33b4c625 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:50.514384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1235453-2e8a-4618-a438-709d33b4c625 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:50.578333 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c1235453-2e8a-4618-a438-709d33b4c625 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:50.579652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c1235453-2e8a-4618-a438-709d33b4c625 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:50.595670 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 861/1730] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:50 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:18:50.669902 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:50.720816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 870/1731] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:50.728165 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b70fc7-d63e-4cff-9952-3350790c534e None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:50.728617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b70fc7-d63e-4cff-9952-3350790c534e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:50.728994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b70fc7-d63e-4cff-9952-3350790c534e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:50.730705 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f53c33-b073-42a2-9a84-a8279e1ee924 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:50.731042 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f53c33-b073-42a2-9a84-a8279e1ee924 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:50.731323 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f53c33-b073-42a2-9a84-a8279e1ee924 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:50.733468 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 871/1732] 10.222.0.15 () {54 vars in 912 bytes} [Tue Sep 13 01:18:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:18:50.740231 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8f5382-52d6-424a-84fd-7b0053fc4740 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:50.740691 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8f5382-52d6-424a-84fd-7b0053fc4740 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:50.740826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8f5382-52d6-424a-84fd-7b0053fc4740 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:50.744005 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 872/1733] 10.222.0.15 () {54 vars in 912 bytes} [Tue Sep 13 01:18:50 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:18:50.745950 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edb34ea-61c3-4e03-82ef-491552e7e362 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:50.745950 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edb34ea-61c3-4e03-82ef-491552e7e362 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:50.746121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edb34ea-61c3-4e03-82ef-491552e7e362 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:50.776073 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-30b70fc7-d63e-4cff-9952-3350790c534e None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:50.777375 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-30b70fc7-d63e-4cff-9952-3350790c534e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:50.797480 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 862/1734] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:50.798976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689bad38-bfd5-448d-a40e-d9b76cd2ae81 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:50.799327 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689bad38-bfd5-448d-a40e-d9b76cd2ae81 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:50.799614 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689bad38-bfd5-448d-a40e-d9b76cd2ae81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:50.827200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8edb34ea-61c3-4e03-82ef-491552e7e362 None None] MFA Rules not processed for user `09cd263f72484733adaaa0850ebfa609`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:50.829055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8edb34ea-61c3-4e03-82ef-491552e7e362 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:50.866768 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-689bad38-bfd5-448d-a40e-d9b76cd2ae81 None None] MFA Rules not processed for user `09cd263f72484733adaaa0850ebfa609`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:50.878965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-689bad38-bfd5-448d-a40e-d9b76cd2ae81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:50.967650 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 873/1735] 10.222.0.15 () {58 vars in 1009 bytes} [Tue Sep 13 01:18:50 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:18:50.970322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edb34ea-61c3-4e03-82ef-491552e7e362 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:51.001523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edb34ea-61c3-4e03-82ef-491552e7e362 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:51.013702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4d9978-698c-4dd6-8c7a-2dbd149f3074 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:51.014891 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4d9978-698c-4dd6-8c7a-2dbd149f3074 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:51.015212 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4d9978-698c-4dd6-8c7a-2dbd149f3074 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:51.015499 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4d9978-698c-4dd6-8c7a-2dbd149f3074 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:51.016182 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4d9978-698c-4dd6-8c7a-2dbd149f3074 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:18:51.016494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4d9978-698c-4dd6-8c7a-2dbd149f3074 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:51.018116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4d9978-698c-4dd6-8c7a-2dbd149f3074 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:51.022224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 863/1736] 10.222.0.15 () {58 vars in 1009 bytes} [Tue Sep 13 01:18:50 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:51.024832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689bad38-bfd5-448d-a40e-d9b76cd2ae81 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:51.030195 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-689bad38-bfd5-448d-a40e-d9b76cd2ae81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:51.040257 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 874/1737] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:18:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:18:51.045009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4d9978-698c-4dd6-8c7a-2dbd149f3074 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:51.059501 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689bad38-bfd5-448d-a40e-d9b76cd2ae81 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:51.064905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4d9978-698c-4dd6-8c7a-2dbd149f3074 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:51.070144 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91df23a7-7904-4ecb-a168-46172039a5dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:51.071076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91df23a7-7904-4ecb-a168-46172039a5dd admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:51.071076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91df23a7-7904-4ecb-a168-46172039a5dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:51.071357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91df23a7-7904-4ecb-a168-46172039a5dd admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:51.071987 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91df23a7-7904-4ecb-a168-46172039a5dd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:51.072757 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a5f202-d0b6-465b-81e7-32eafa32e460 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:51.073395 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a5f202-d0b6-465b-81e7-32eafa32e460 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:51.073675 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a5f202-d0b6-465b-81e7-32eafa32e460 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:51.073950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a5f202-d0b6-465b-81e7-32eafa32e460 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:51.074054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91df23a7-7904-4ecb-a168-46172039a5dd admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:51.099415 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a5f202-d0b6-465b-81e7-32eafa32e460 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:51.100910 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a5f202-d0b6-465b-81e7-32eafa32e460 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:51.122758 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 864/1738] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:51 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:51.124262 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a5f202-d0b6-465b-81e7-32eafa32e460 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:51.128303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e4a5f202-d0b6-465b-81e7-32eafa32e460 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:51.129214 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 875/1739] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:51 2022] POST /identity/v3/projects => generated 365 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:51.134261 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91df23a7-7904-4ecb-a168-46172039a5dd admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:51.154649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a5f202-d0b6-465b-81e7-32eafa32e460 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:51.260210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91df23a7-7904-4ecb-a168-46172039a5dd admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:51.270412 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a523e9f-fcd5-4aa1-b342-4acfb5c799af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:51.271907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a523e9f-fcd5-4aa1-b342-4acfb5c799af admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:51.272260 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a523e9f-fcd5-4aa1-b342-4acfb5c799af admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:51.272548 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a523e9f-fcd5-4aa1-b342-4acfb5c799af admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:51.273223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a523e9f-fcd5-4aa1-b342-4acfb5c799af admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:51.275762 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a523e9f-fcd5-4aa1-b342-4acfb5c799af admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:51.291883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb7acd8-e318-45bb-85f6-62c234d1c022 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:51.292998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb7acd8-e318-45bb-85f6-62c234d1c022 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:51.294058 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb7acd8-e318-45bb-85f6-62c234d1c022 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:51.294058 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb7acd8-e318-45bb-85f6-62c234d1c022 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:51.318314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 876/1740] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:51 2022] POST /identity/v3/users => generated 283 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:51.320105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a523e9f-fcd5-4aa1-b342-4acfb5c799af admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:51.321677 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb7acd8-e318-45bb-85f6-62c234d1c022 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:51.324135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1a523e9f-fcd5-4aa1-b342-4acfb5c799af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:51.326119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb7acd8-e318-45bb-85f6-62c234d1c022 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:51.342688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 865/1741] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:51 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 219 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:51.344421 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb7acd8-e318-45bb-85f6-62c234d1c022 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:51.354326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a523e9f-fcd5-4aa1-b342-4acfb5c799af admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:51.360911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a9f52b-3a80-414f-a870-d2b99afa3040 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:51.362049 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a9f52b-3a80-414f-a870-d2b99afa3040 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:51.362334 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a9f52b-3a80-414f-a870-d2b99afa3040 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:51.363259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a9f52b-3a80-414f-a870-d2b99afa3040 admin admin] PATH_INFO: `/v3/projects/550365774da34580a8a68ff7b2134a7b` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:51.364591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a9f52b-3a80-414f-a870-d2b99afa3040 admin admin] RBAC: Authorizing `identity:delete_project(project_id=550365774da34580a8a68ff7b2134a7b)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:51.367142 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a9f52b-3a80-414f-a870-d2b99afa3040 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:51.369724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb7acd8-e318-45bb-85f6-62c234d1c022 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:51.382286 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dffabe-f832-4fc9-8293-c1a8f70ef9da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:51.383053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dffabe-f832-4fc9-8293-c1a8f70ef9da admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:51.383410 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dffabe-f832-4fc9-8293-c1a8f70ef9da admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:51.383756 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dffabe-f832-4fc9-8293-c1a8f70ef9da admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:51.385129 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6dffabe-f832-4fc9-8293-c1a8f70ef9da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:51.387125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6dffabe-f832-4fc9-8293-c1a8f70ef9da admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:51.401366 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 866/1742] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:51 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:51.405725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dffabe-f832-4fc9-8293-c1a8f70ef9da admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:51.420628 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dffabe-f832-4fc9-8293-c1a8f70ef9da admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:51.426368 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0b23c8-4603-46d4-8b27-65e0aa38c9a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:51.427209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0b23c8-4603-46d4-8b27-65e0aa38c9a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:51.427603 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0b23c8-4603-46d4-8b27-65e0aa38c9a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:51.428007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0b23c8-4603-46d4-8b27-65e0aa38c9a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:51.428765 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0b23c8-4603-46d4-8b27-65e0aa38c9a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:51.430620 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0b23c8-4603-46d4-8b27-65e0aa38c9a6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:51.453610 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 867/1743] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:51 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:18:51.458587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0b23c8-4603-46d4-8b27-65e0aa38c9a6 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:51.465355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:51.472404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0b23c8-4603-46d4-8b27-65e0aa38c9a6 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:51.473663 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-06a9f52b-3a80-414f-a870-d2b99afa3040 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '550365774da34580a8a68ff7b2134a7b', 'request_id': 'req-06a9f52b-3a80-414f-a870-d2b99afa3040'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:51.488486 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-06a9f52b-3a80-414f-a870-d2b99afa3040 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '550365774da34580a8a68ff7b2134a7b', 'request_id': 'req-06a9f52b-3a80-414f-a870-d2b99afa3040'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:51.494126 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:51.502373 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-06a9f52b-3a80-414f-a870-d2b99afa3040 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '550365774da34580a8a68ff7b2134a7b', 'request_id': 'req-06a9f52b-3a80-414f-a870-d2b99afa3040'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:51.540018 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-06a9f52b-3a80-414f-a870-d2b99afa3040 admin admin] The token cache is being invalidate because project 550365774da34580a8a68ff7b2134a7b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:18:51.541465 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-06a9f52b-3a80-414f-a870-d2b99afa3040 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:51.544697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-06a9f52b-3a80-414f-a870-d2b99afa3040 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:51.549512 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 877/1744] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:18:51 2022] DELETE /identity/v3/projects/550365774da34580a8a68ff7b2134a7b => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:51.555384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a9f52b-3a80-414f-a870-d2b99afa3040 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:51.558698 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-06a9f52b-3a80-414f-a870-d2b99afa3040 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:51.582061 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59eb7920-82cc-4ffc-beb9-034737acf582 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:51.582743 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59eb7920-82cc-4ffc-beb9-034737acf582 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:51.583275 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59eb7920-82cc-4ffc-beb9-034737acf582 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:51.583958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59eb7920-82cc-4ffc-beb9-034737acf582 admin admin] PATH_INFO: `/v3/projects/b3d92f0132c84f26b55b2a75a4328955/users/26198591c6774b4c87b5916420bbb47d/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:51.617510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59eb7920-82cc-4ffc-beb9-034737acf582 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3d92f0132c84f26b55b2a75a4328955, user_id=26198591c6774b4c87b5916420bbb47d, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:51.620199 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59eb7920-82cc-4ffc-beb9-034737acf582 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:51.672214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 868/1745] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:51 2022] PUT /identity/v3/projects/b3d92f0132c84f26b55b2a75a4328955/users/26198591c6774b4c87b5916420bbb47d/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:51.673761 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b7fdf4-7330-4bbe-b94d-e3ef68710265 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:51.674072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b7fdf4-7330-4bbe-b94d-e3ef68710265 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:51.674355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b7fdf4-7330-4bbe-b94d-e3ef68710265 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:51.725936 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a9f52b-3a80-414f-a870-d2b99afa3040 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:51.727048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-89b7fdf4-7330-4bbe-b94d-e3ef68710265 None None] MFA Rules not processed for user `26198591c6774b4c87b5916420bbb47d`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:51.729103 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-89b7fdf4-7330-4bbe-b94d-e3ef68710265 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:51.888199 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3c557d-f2e7-4517-904f-946ed721243e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:51.888988 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3c557d-f2e7-4517-904f-946ed721243e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:51.889265 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3c557d-f2e7-4517-904f-946ed721243e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:51.890036 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3c557d-f2e7-4517-904f-946ed721243e admin admin] PATH_INFO: `/v3/users/d92ce2186fe443749ff13ad7e90bcb1f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:51.892026 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3c557d-f2e7-4517-904f-946ed721243e admin admin] RBAC: Authorizing `identity:delete_user(user_id=d92ce2186fe443749ff13ad7e90bcb1f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:51.894050 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3c557d-f2e7-4517-904f-946ed721243e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:51.907672 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 869/1746] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:51 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:51.908526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b7fdf4-7330-4bbe-b94d-e3ef68710265 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:51.909805 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-89b7fdf4-7330-4bbe-b94d-e3ef68710265 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:51.936718 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b7fdf4-7330-4bbe-b94d-e3ef68710265 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:51.958034 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf8fcea-67a8-465c-9b63-48c3924d818a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:51.958695 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf8fcea-67a8-465c-9b63-48c3924d818a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:51.959016 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf8fcea-67a8-465c-9b63-48c3924d818a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:51.959288 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf8fcea-67a8-465c-9b63-48c3924d818a admin admin] PATH_INFO: `/v3/projects/bc1f33ab6f9e4319ad62cd516201ecfa` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:51.967791 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf8fcea-67a8-465c-9b63-48c3924d818a admin admin] RBAC: Authorizing `identity:delete_project(project_id=bc1f33ab6f9e4319ad62cd516201ecfa)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:51.969817 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf8fcea-67a8-465c-9b63-48c3924d818a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:52.046541 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9b3c557d-f2e7-4517-904f-946ed721243e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd92ce2186fe443749ff13ad7e90bcb1f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:52.047662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9b3c557d-f2e7-4517-904f-946ed721243e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd92ce2186fe443749ff13ad7e90bcb1f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:52.073737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9b3c557d-f2e7-4517-904f-946ed721243e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd92ce2186fe443749ff13ad7e90bcb1f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:52.076399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9b3c557d-f2e7-4517-904f-946ed721243e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd92ce2186fe443749ff13ad7e90bcb1f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:52.114293 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cdf8fcea-67a8-465c-9b63-48c3924d818a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bc1f33ab6f9e4319ad62cd516201ecfa', 'request_id': 'req-cdf8fcea-67a8-465c-9b63-48c3924d818a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:52.133337 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9b3c557d-f2e7-4517-904f-946ed721243e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd92ce2186fe443749ff13ad7e90bcb1f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:52.139660 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cdf8fcea-67a8-465c-9b63-48c3924d818a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bc1f33ab6f9e4319ad62cd516201ecfa', 'request_id': 'req-cdf8fcea-67a8-465c-9b63-48c3924d818a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:52.146427 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:52.155312 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 878/1747] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:18:51 2022] DELETE /identity/v3/users/d92ce2186fe443749ff13ad7e90bcb1f => generated 0 bytes in 602 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:18:52.158415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3c557d-f2e7-4517-904f-946ed721243e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:52.171760 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cdf8fcea-67a8-465c-9b63-48c3924d818a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bc1f33ab6f9e4319ad62cd516201ecfa', 'request_id': 'req-cdf8fcea-67a8-465c-9b63-48c3924d818a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:52.176688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9b3c557d-f2e7-4517-904f-946ed721243e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:52.211981 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cdf8fcea-67a8-465c-9b63-48c3924d818a admin admin] The token cache is being invalidate because project bc1f33ab6f9e4319ad62cd516201ecfa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:18:52.212388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cdf8fcea-67a8-465c-9b63-48c3924d818a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:52.221004 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cdf8fcea-67a8-465c-9b63-48c3924d818a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:52.228391 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 870/1748] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:18:51 2022] DELETE /identity/v3/projects/bc1f33ab6f9e4319ad62cd516201ecfa => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:52.228391 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf8fcea-67a8-465c-9b63-48c3924d818a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:52.230443 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cdf8fcea-67a8-465c-9b63-48c3924d818a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:52.280679 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:52.393515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3c557d-f2e7-4517-904f-946ed721243e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:52.406353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a62d056-7b7e-4299-8068-da077ddb5b32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:52.407211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a62d056-7b7e-4299-8068-da077ddb5b32 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:52.407500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a62d056-7b7e-4299-8068-da077ddb5b32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:52.407772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a62d056-7b7e-4299-8068-da077ddb5b32 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:52.409133 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a62d056-7b7e-4299-8068-da077ddb5b32 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:52.414391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a62d056-7b7e-4299-8068-da077ddb5b32 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:52.445900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf8fcea-67a8-465c-9b63-48c3924d818a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:52.487128 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 879/1749] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:52 2022] POST /identity/v3/projects => generated 363 bytes in 331 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:52.488610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b027edbf-26a2-49b8-95d1-cf5b98a8a3bf None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:52.488902 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b027edbf-26a2-49b8-95d1-cf5b98a8a3bf None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:52.489184 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b027edbf-26a2-49b8-95d1-cf5b98a8a3bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:52.541747 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b027edbf-26a2-49b8-95d1-cf5b98a8a3bf None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:52.543264 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b027edbf-26a2-49b8-95d1-cf5b98a8a3bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:52.664382 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e550e688-ad19-4ec8-a7c7-88122924de9c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:52.665151 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e550e688-ad19-4ec8-a7c7-88122924de9c service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:52.665626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e550e688-ad19-4ec8-a7c7-88122924de9c service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:52.665766 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e550e688-ad19-4ec8-a7c7-88122924de9c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:52.668445 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e550e688-ad19-4ec8-a7c7-88122924de9c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:52.733271 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 880/1750] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:52.734563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b027edbf-26a2-49b8-95d1-cf5b98a8a3bf None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:52.737000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b027edbf-26a2-49b8-95d1-cf5b98a8a3bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:52.800482 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e550e688-ad19-4ec8-a7c7-88122924de9c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:52.802276 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e550e688-ad19-4ec8-a7c7-88122924de9c service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:52.813505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 871/1751] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:52 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 587 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:52.815068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e797226-999e-4c78-86a3-174b190a496f None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:52.815870 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e797226-999e-4c78-86a3-174b190a496f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:52.815870 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e797226-999e-4c78-86a3-174b190a496f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:52.857507 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-0e797226-999e-4c78-86a3-174b190a496f None None] MFA Rules not processed for user `06797d4f56954444b3317f2339f020c4`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:52.860079 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0e797226-999e-4c78-86a3-174b190a496f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:52.947311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b027edbf-26a2-49b8-95d1-cf5b98a8a3bf None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:52.956206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:52.963648 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd962fc7-947d-4c8d-8585-b376387b663b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8eaff1b843a145a49994dcf089f44315', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:52.964369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd962fc7-947d-4c8d-8585-b376387b663b service placement] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:52.964779 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd962fc7-947d-4c8d-8585-b376387b663b service placement] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:52.964989 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd962fc7-947d-4c8d-8585-b376387b663b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:52.971462 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-cd962fc7-947d-4c8d-8585-b376387b663b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:52.990045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd962fc7-947d-4c8d-8585-b376387b663b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:52.992157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd962fc7-947d-4c8d-8585-b376387b663b service placement] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:53.019094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 881/1752] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:53.020859 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 872/1753] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:52 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:53.021271 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3617b0d1-0f44-4b78-a268-08d78e5f345d None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:53.021768 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3617b0d1-0f44-4b78-a268-08d78e5f345d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:53.022406 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3617b0d1-0f44-4b78-a268-08d78e5f345d None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:53.027723 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 882/1754] 10.222.0.15 () {54 vars in 961 bytes} [Tue Sep 13 01:18:53 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:18:53.029362 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e797226-999e-4c78-86a3-174b190a496f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:53.029821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a0316c-4473-4312-a9e1-369904546108 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:53.030456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a0316c-4473-4312-a9e1-369904546108 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:53.031536 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a0316c-4473-4312-a9e1-369904546108 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:53.032568 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0e797226-999e-4c78-86a3-174b190a496f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:53.058828 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-21a0316c-4473-4312-a9e1-369904546108 None None] MFA Rules not processed for user `0c5dcfa5794843a5896fa051a4554e1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:53.060995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-21a0316c-4473-4312-a9e1-369904546108 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:53.072634 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:53.080110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e797226-999e-4c78-86a3-174b190a496f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:53.098678 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e93de7-33c7-489c-958e-70dd12405429 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:53.103027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e93de7-33c7-489c-958e-70dd12405429 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:53.103309 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e93de7-33c7-489c-958e-70dd12405429 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:53.103585 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e93de7-33c7-489c-958e-70dd12405429 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:53.104592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e93de7-33c7-489c-958e-70dd12405429 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:53.111133 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e93de7-33c7-489c-958e-70dd12405429 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:53.177195 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 873/1755] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:53 2022] POST /identity/v3/users => generated 288 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:53.182214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7c7d4d-7d73-4bca-b9fb-08d3e42be902 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:53.182214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7c7d4d-7d73-4bca-b9fb-08d3e42be902 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:53.189724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 883/1756] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:53 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:53.189724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a0316c-4473-4312-a9e1-369904546108 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:53.192757 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-21a0316c-4473-4312-a9e1-369904546108 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:53.193671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7c7d4d-7d73-4bca-b9fb-08d3e42be902 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:53.225269 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a0316c-4473-4312-a9e1-369904546108 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:53.232441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-5d7c7d4d-7d73-4bca-b9fb-08d3e42be902 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:53.234604 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5d7c7d4d-7d73-4bca-b9fb-08d3e42be902 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:53.241027 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766b9bc4-dbbf-42c2-8e0b-ad569b509020 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:53.241825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766b9bc4-dbbf-42c2-8e0b-ad569b509020 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:53.242222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766b9bc4-dbbf-42c2-8e0b-ad569b509020 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:53.242621 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766b9bc4-dbbf-42c2-8e0b-ad569b509020 admin admin] PATH_INFO: `/v3/projects/9aaf62e0d3054a13b78ec1377771699f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:53.244879 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766b9bc4-dbbf-42c2-8e0b-ad569b509020 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9aaf62e0d3054a13b78ec1377771699f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:53.245977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 874/1757] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:53.247520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5700e1b4-6654-4776-8f5a-5d69b99a2c40 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:53.247807 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5700e1b4-6654-4776-8f5a-5d69b99a2c40 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:53.248113 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766b9bc4-dbbf-42c2-8e0b-ad569b509020 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:53.248227 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5700e1b4-6654-4776-8f5a-5d69b99a2c40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:53.292716 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-5700e1b4-6654-4776-8f5a-5d69b99a2c40 None None] MFA Rules not processed for user `8eaff1b843a145a49994dcf089f44315`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:53.297665 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5700e1b4-6654-4776-8f5a-5d69b99a2c40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:53.356057 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 875/1758] 10.222.0.15 () {58 vars in 1058 bytes} [Tue Sep 13 01:18:53 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:53.357500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70efc3f-79b2-44cc-88a5-2054b561a2d7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:53.357843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70efc3f-79b2-44cc-88a5-2054b561a2d7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:53.358297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70efc3f-79b2-44cc-88a5-2054b561a2d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:53.450689 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-766b9bc4-dbbf-42c2-8e0b-ad569b509020 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9aaf62e0d3054a13b78ec1377771699f', 'request_id': 'req-766b9bc4-dbbf-42c2-8e0b-ad569b509020'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:53.464600 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c70efc3f-79b2-44cc-88a5-2054b561a2d7 None None] MFA Rules not processed for user `06797d4f56954444b3317f2339f020c4`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:53.464600 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c70efc3f-79b2-44cc-88a5-2054b561a2d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:53.516088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-766b9bc4-dbbf-42c2-8e0b-ad569b509020 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9aaf62e0d3054a13b78ec1377771699f', 'request_id': 'req-766b9bc4-dbbf-42c2-8e0b-ad569b509020'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:53.541337 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-766b9bc4-dbbf-42c2-8e0b-ad569b509020 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9aaf62e0d3054a13b78ec1377771699f', 'request_id': 'req-766b9bc4-dbbf-42c2-8e0b-ad569b509020'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:53.575504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-766b9bc4-dbbf-42c2-8e0b-ad569b509020 admin admin] The token cache is being invalidate because project 9aaf62e0d3054a13b78ec1377771699f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:18:53.577239 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-766b9bc4-dbbf-42c2-8e0b-ad569b509020 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:53.580196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-766b9bc4-dbbf-42c2-8e0b-ad569b509020 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:18:53.588646 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 884/1759] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:18:53 2022] DELETE /identity/v3/projects/9aaf62e0d3054a13b78ec1377771699f => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:53.588646 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766b9bc4-dbbf-42c2-8e0b-ad569b509020 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:53.590608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-766b9bc4-dbbf-42c2-8e0b-ad569b509020 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:53.679584 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 876/1760] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:53 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 323 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:53.680797 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70efc3f-79b2-44cc-88a5-2054b561a2d7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:53.683786 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c70efc3f-79b2-44cc-88a5-2054b561a2d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:53.759452 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766b9bc4-dbbf-42c2-8e0b-ad569b509020 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:53.782728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdaab01-e0b4-4573-9473-5d4c11d65719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:53.784117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdaab01-e0b4-4573-9473-5d4c11d65719 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:53.786685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdaab01-e0b4-4573-9473-5d4c11d65719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:53.787040 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdaab01-e0b4-4573-9473-5d4c11d65719 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:53.790315 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbdaab01-e0b4-4573-9473-5d4c11d65719 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:53.795874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbdaab01-e0b4-4573-9473-5d4c11d65719 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:53.833608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 885/1761] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:53 2022] GET /identity/v3/roles => generated 1420 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:53.839409 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdaab01-e0b4-4573-9473-5d4c11d65719 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:53.850019 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-fbdaab01-e0b4-4573-9473-5d4c11d65719 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:53.857597 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70efc3f-79b2-44cc-88a5-2054b561a2d7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:53.873720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56328b14-df29-4f30-8bf1-b05f55d8aadf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:53.876560 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56328b14-df29-4f30-8bf1-b05f55d8aadf admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:53.877102 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56328b14-df29-4f30-8bf1-b05f55d8aadf admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:53.877384 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56328b14-df29-4f30-8bf1-b05f55d8aadf admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:53.879584 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56328b14-df29-4f30-8bf1-b05f55d8aadf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:18:53.879907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56328b14-df29-4f30-8bf1-b05f55d8aadf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:53.885767 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56328b14-df29-4f30-8bf1-b05f55d8aadf admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:53.921911 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 877/1762] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:18:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:18:53.924297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56328b14-df29-4f30-8bf1-b05f55d8aadf admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:53.928411 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-56328b14-df29-4f30-8bf1-b05f55d8aadf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:54.006156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:54.055535 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdaab01-e0b4-4573-9473-5d4c11d65719 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:54.070718 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807b584c-341f-4a39-8e6e-a0f8cfa046a7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8eaff1b843a145a49994dcf089f44315', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:54.071325 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807b584c-341f-4a39-8e6e-a0f8cfa046a7 service placement] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:54.071604 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807b584c-341f-4a39-8e6e-a0f8cfa046a7 service placement] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:54.071872 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807b584c-341f-4a39-8e6e-a0f8cfa046a7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:54.074746 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-807b584c-341f-4a39-8e6e-a0f8cfa046a7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:54.078401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:54.088780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56328b14-df29-4f30-8bf1-b05f55d8aadf admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:54.100097 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcc5164-2357-4422-92b9-599601f53696 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:54.101176 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcc5164-2357-4422-92b9-599601f53696 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:54.101466 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcc5164-2357-4422-92b9-599601f53696 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:54.101742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcc5164-2357-4422-92b9-599601f53696 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:54.106482 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6bcc5164-2357-4422-92b9-599601f53696 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:54.107343 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807b584c-341f-4a39-8e6e-a0f8cfa046a7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:54.111557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807b584c-341f-4a39-8e6e-a0f8cfa046a7 service placement] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:54.137087 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 886/1763] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:53 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:54.149994 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6539a0d-a976-4afe-a70e-f9d930251e5f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:54.150281 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6539a0d-a976-4afe-a70e-f9d930251e5f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:54.150557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6539a0d-a976-4afe-a70e-f9d930251e5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:54.227375 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b6539a0d-a976-4afe-a70e-f9d930251e5f None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:54.228076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b6539a0d-a976-4afe-a70e-f9d930251e5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:54.261818 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 887/1764] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:54.263055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6539a0d-a976-4afe-a70e-f9d930251e5f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:54.266334 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b6539a0d-a976-4afe-a70e-f9d930251e5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:54.346479 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bcc5164-2357-4422-92b9-599601f53696 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:54.348178 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bcc5164-2357-4422-92b9-599601f53696 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:54.414905 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 878/1765] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:53 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 492 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:54.417999 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcc5164-2357-4422-92b9-599601f53696 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:54.457890 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcc5164-2357-4422-92b9-599601f53696 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:54.493981 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad830450-ffe3-4b68-89f5-11e8ef1199b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:54.497208 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad830450-ffe3-4b68-89f5-11e8ef1199b7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:54.497489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad830450-ffe3-4b68-89f5-11e8ef1199b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:54.497761 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad830450-ffe3-4b68-89f5-11e8ef1199b7 admin admin] PATH_INFO: `/v3/projects/9b4cfdf1aede49af91f83f67637e4722/users/8852beeb168442e4b99996e2a8189937/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:54.529096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad830450-ffe3-4b68-89f5-11e8ef1199b7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b4cfdf1aede49af91f83f67637e4722, user_id=8852beeb168442e4b99996e2a8189937, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:54.533521 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad830450-ffe3-4b68-89f5-11e8ef1199b7 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:54.563100 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 879/1766] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:54 2022] PUT /identity/v3/projects/9b4cfdf1aede49af91f83f67637e4722/users/8852beeb168442e4b99996e2a8189937/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:54.564759 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad830450-ffe3-4b68-89f5-11e8ef1199b7 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:54.602043 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6539a0d-a976-4afe-a70e-f9d930251e5f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:54.831116 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad830450-ffe3-4b68-89f5-11e8ef1199b7 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:54.843803 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcf14d5-d0a3-4084-9fda-58fb8b75f0a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:54.848356 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcf14d5-d0a3-4084-9fda-58fb8b75f0a7 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:54.848568 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcf14d5-d0a3-4084-9fda-58fb8b75f0a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:54.849144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcf14d5-d0a3-4084-9fda-58fb8b75f0a7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:54.849492 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcf14d5-d0a3-4084-9fda-58fb8b75f0a7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:54.856831 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcf14d5-d0a3-4084-9fda-58fb8b75f0a7 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:54.867488 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839afa69-e50c-424a-9720-fcd7f6cc133f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:54.868119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839afa69-e50c-424a-9720-fcd7f6cc133f service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:54.868395 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839afa69-e50c-424a-9720-fcd7f6cc133f service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:54.868663 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839afa69-e50c-424a-9720-fcd7f6cc133f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:54.885244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-839afa69-e50c-424a-9720-fcd7f6cc133f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:54.996725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 880/1767] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:54 2022] POST /identity/v3/projects => generated 355 bytes in 433 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:55.001210 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab20da27-94ca-4f15-a247-05ed54fb0e8e None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:55.005411 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab20da27-94ca-4f15-a247-05ed54fb0e8e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:55.005886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab20da27-94ca-4f15-a247-05ed54fb0e8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:55.060169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ab20da27-94ca-4f15-a247-05ed54fb0e8e None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:55.062568 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ab20da27-94ca-4f15-a247-05ed54fb0e8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:55.132457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:55.140066 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839afa69-e50c-424a-9720-fcd7f6cc133f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:55.159344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839afa69-e50c-424a-9720-fcd7f6cc133f service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:55.230356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 888/1768] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:54 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 966 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:55.231413 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839afa69-e50c-424a-9720-fcd7f6cc133f service nova] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:55.260707 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:55.286592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 881/1769] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 289 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:55.288775 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab20da27-94ca-4f15-a247-05ed54fb0e8e None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:55.310385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab20da27-94ca-4f15-a247-05ed54fb0e8e None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:55.319004 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76640e43-1629-458b-ae21-9f2bae774fd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:55.319743 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76640e43-1629-458b-ae21-9f2bae774fd8 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:55.320123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76640e43-1629-458b-ae21-9f2bae774fd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:55.320509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76640e43-1629-458b-ae21-9f2bae774fd8 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:55.321403 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76640e43-1629-458b-ae21-9f2bae774fd8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:55.328801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839afa69-e50c-424a-9720-fcd7f6cc133f service nova] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:55.351497 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76640e43-1629-458b-ae21-9f2bae774fd8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:55.363754 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb63cae-6a3c-443a-9cee-ea3aeb90f0d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:55.375795 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb63cae-6a3c-443a-9cee-ea3aeb90f0d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:55.376090 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb63cae-6a3c-443a-9cee-ea3aeb90f0d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:55.376374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb63cae-6a3c-443a-9cee-ea3aeb90f0d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:55.377063 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb63cae-6a3c-443a-9cee-ea3aeb90f0d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:55.404358 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb63cae-6a3c-443a-9cee-ea3aeb90f0d7 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:55.408744 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 882/1770] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:55 2022] POST /identity/v3/users => generated 278 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:55.412855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76640e43-1629-458b-ae21-9f2bae774fd8 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:55.435698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76640e43-1629-458b-ae21-9f2bae774fd8 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:55.442580 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d76def-0da1-4423-b120-e9a0f6042126 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:55.443709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d76def-0da1-4423-b120-e9a0f6042126 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:55.444259 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d76def-0da1-4423-b120-e9a0f6042126 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:55.444746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d76def-0da1-4423-b120-e9a0f6042126 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:55.445561 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d76def-0da1-4423-b120-e9a0f6042126 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:18:55.446105 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d76def-0da1-4423-b120-e9a0f6042126 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:55.447169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 889/1771] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:55 2022] GET /identity/v3/roles => generated 1420 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:55.448141 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb63cae-6a3c-443a-9cee-ea3aeb90f0d7 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:55.451357 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d76def-0da1-4423-b120-e9a0f6042126 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:55.486268 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 883/1772] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:18:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:18:55.488262 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d76def-0da1-4423-b120-e9a0f6042126 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:55.513585 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d76def-0da1-4423-b120-e9a0f6042126 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:55.517532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb63cae-6a3c-443a-9cee-ea3aeb90f0d7 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:55.525263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0890e7-0e00-4990-8648-5184db3bb83c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:55.528267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0890e7-0e00-4990-8648-5184db3bb83c admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:55.533500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0890e7-0e00-4990-8648-5184db3bb83c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:55.533500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0890e7-0e00-4990-8648-5184db3bb83c admin admin] PATH_INFO: `/v3/projects/9b4cfdf1aede49af91f83f67637e4722/users/8852beeb168442e4b99996e2a8189937/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:55.535885 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c856388-3697-4b57-b0d6-65795c86ca01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:55.536508 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c856388-3697-4b57-b0d6-65795c86ca01 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:55.538233 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c856388-3697-4b57-b0d6-65795c86ca01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:55.538510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c856388-3697-4b57-b0d6-65795c86ca01 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:55.538967 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0890e7-0e00-4990-8648-5184db3bb83c admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b4cfdf1aede49af91f83f67637e4722, user_id=8852beeb168442e4b99996e2a8189937, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:55.540814 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c856388-3697-4b57-b0d6-65795c86ca01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:55.554841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c856388-3697-4b57-b0d6-65795c86ca01 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:55.557117 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0890e7-0e00-4990-8648-5184db3bb83c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:55.598965 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 890/1773] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:55 2022] GET /identity/v3/roles => generated 1420 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:55.602522 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c856388-3697-4b57-b0d6-65795c86ca01 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:55.605911 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 884/1774] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:55 2022] PUT /identity/v3/projects/9b4cfdf1aede49af91f83f67637e4722/users/8852beeb168442e4b99996e2a8189937/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:55.610467 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0890e7-0e00-4990-8648-5184db3bb83c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:55.654604 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c856388-3697-4b57-b0d6-65795c86ca01 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:55.974882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0890e7-0e00-4990-8648-5184db3bb83c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:55.990439 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5157dc-f715-4f0f-8f60-5bf9dd3780bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:55.991340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5157dc-f715-4f0f-8f60-5bf9dd3780bf admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:55.991760 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5157dc-f715-4f0f-8f60-5bf9dd3780bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:55.992789 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5157dc-f715-4f0f-8f60-5bf9dd3780bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:55.992789 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5157dc-f715-4f0f-8f60-5bf9dd3780bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:55.996315 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5157dc-f715-4f0f-8f60-5bf9dd3780bf admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:56.010970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874c8c6b-81d0-4500-b690-b8239628a9f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:56.011670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874c8c6b-81d0-4500-b690-b8239628a9f2 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:56.014533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874c8c6b-81d0-4500-b690-b8239628a9f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:56.014903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874c8c6b-81d0-4500-b690-b8239628a9f2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:56.015670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874c8c6b-81d0-4500-b690-b8239628a9f2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:56.021994 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874c8c6b-81d0-4500-b690-b8239628a9f2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:56.029453 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 885/1775] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:55 2022] GET /identity/v3/roles => generated 1420 bytes in 422 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:56.031753 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23d5c98-b2af-4ef3-a8bf-1edfe8e9cc47 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:56.034050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23d5c98-b2af-4ef3-a8bf-1edfe8e9cc47 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:56.035255 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23d5c98-b2af-4ef3-a8bf-1edfe8e9cc47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:56.099228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 891/1776] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:55 2022] POST /identity/v3/projects => generated 405 bytes in 500 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:56.100284 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874c8c6b-81d0-4500-b690-b8239628a9f2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:56.127361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d23d5c98-b2af-4ef3-a8bf-1edfe8e9cc47 None None] MFA Rules not processed for user `8852beeb168442e4b99996e2a8189937`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:56.134871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d23d5c98-b2af-4ef3-a8bf-1edfe8e9cc47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:56.240308 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874c8c6b-81d0-4500-b690-b8239628a9f2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:56.260977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c5963e-b20e-4515-9f01-a05ea683ee2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:56.264278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c5963e-b20e-4515-9f01-a05ea683ee2a admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:56.264587 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c5963e-b20e-4515-9f01-a05ea683ee2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:56.264884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c5963e-b20e-4515-9f01-a05ea683ee2a admin admin] PATH_INFO: `/v3/projects/8b7ef23e634648bcb97f0ce19114b8f4/users/1b1c2bb130294b969d21071c9b93ea09/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:56.289113 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 886/1777] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:56 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:56.291495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23d5c98-b2af-4ef3-a8bf-1edfe8e9cc47 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:56.299773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c5963e-b20e-4515-9f01-a05ea683ee2a admin admin] RBAC: Authorizing `identity:create_grant(project_id=8b7ef23e634648bcb97f0ce19114b8f4, user_id=1b1c2bb130294b969d21071c9b93ea09, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:56.302529 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c5963e-b20e-4515-9f01-a05ea683ee2a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:56.324570 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23d5c98-b2af-4ef3-a8bf-1edfe8e9cc47 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:56.354838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 892/1778] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:56 2022] PUT /identity/v3/projects/8b7ef23e634648bcb97f0ce19114b8f4/users/1b1c2bb130294b969d21071c9b93ea09/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:56.355934 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c5963e-b20e-4515-9f01-a05ea683ee2a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:56.389377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:56.435655 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd5ca3f-e893-463a-bf56-d0839b29e311 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:56.436714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd5ca3f-e893-463a-bf56-d0839b29e311 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:56.437693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd5ca3f-e893-463a-bf56-d0839b29e311 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:56.438079 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd5ca3f-e893-463a-bf56-d0839b29e311 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:56.438830 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd5ca3f-e893-463a-bf56-d0839b29e311 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:56.441098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd5ca3f-e893-463a-bf56-d0839b29e311 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:56.454961 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:56.468668 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c5963e-b20e-4515-9f01-a05ea683ee2a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:56.481562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a0203c-51f8-4239-9319-1bc98d22ff74 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:56.482696 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a0203c-51f8-4239-9319-1bc98d22ff74 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:56.483450 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a0203c-51f8-4239-9319-1bc98d22ff74 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:56.484033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a0203c-51f8-4239-9319-1bc98d22ff74 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:56.510842 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 887/1779] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:56 2022] POST /identity/v3/users => generated 303 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:56.511899 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a64ffa1-e864-46c1-8658-f90b019e7e66 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:56.512303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a64ffa1-e864-46c1-8658-f90b019e7e66 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:56.512695 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a64ffa1-e864-46c1-8658-f90b019e7e66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:56.589767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a0203c-51f8-4239-9319-1bc98d22ff74 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:56.591570 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8a64ffa1-e864-46c1-8658-f90b019e7e66 None None] MFA Rules not processed for user `1b1c2bb130294b969d21071c9b93ea09`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:56.594107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a0203c-51f8-4239-9319-1bc98d22ff74 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:56.596568 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8a64ffa1-e864-46c1-8658-f90b019e7e66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:56.625011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 893/1780] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:56 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 267 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:56.625921 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a0203c-51f8-4239-9319-1bc98d22ff74 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:56.656199 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a0203c-51f8-4239-9319-1bc98d22ff74 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:56.673343 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4e36ed-6b69-42dd-85bf-92815080c0ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:56.673970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4e36ed-6b69-42dd-85bf-92815080c0ff admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:56.674246 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4e36ed-6b69-42dd-85bf-92815080c0ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:56.674514 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4e36ed-6b69-42dd-85bf-92815080c0ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:56.675193 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4e36ed-6b69-42dd-85bf-92815080c0ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:56.677871 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4e36ed-6b69-42dd-85bf-92815080c0ff admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:56.735695 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 894/1781] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:56 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:56.736669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4e36ed-6b69-42dd-85bf-92815080c0ff admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:56.754337 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4e36ed-6b69-42dd-85bf-92815080c0ff admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:56.760042 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fa0f60-5f9c-42b5-929f-44314f7d74cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:56.760839 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fa0f60-5f9c-42b5-929f-44314f7d74cd admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:56.761262 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fa0f60-5f9c-42b5-929f-44314f7d74cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:56.761524 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fa0f60-5f9c-42b5-929f-44314f7d74cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:56.762412 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75fa0f60-5f9c-42b5-929f-44314f7d74cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:56.764079 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75fa0f60-5f9c-42b5-929f-44314f7d74cd admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:56.775592 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 895/1782] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:56 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:56.777290 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fa0f60-5f9c-42b5-929f-44314f7d74cd admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:56.780398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-75fa0f60-5f9c-42b5-929f-44314f7d74cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:56.789418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 888/1783] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:56 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:56.790516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a64ffa1-e864-46c1-8658-f90b019e7e66 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:56.797342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fa0f60-5f9c-42b5-929f-44314f7d74cd admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:56.803668 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95417ba3-2719-43b7-bdd0-0c436ba27c89 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:56.804415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95417ba3-2719-43b7-bdd0-0c436ba27c89 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:56.804708 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95417ba3-2719-43b7-bdd0-0c436ba27c89 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:56.804973 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95417ba3-2719-43b7-bdd0-0c436ba27c89 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:56.807036 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-95417ba3-2719-43b7-bdd0-0c436ba27c89 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:56.807170 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a64ffa1-e864-46c1-8658-f90b019e7e66 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:56.812763 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227d1e54-bf3f-43ab-bbab-0d65e164ed1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:56.813955 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227d1e54-bf3f-43ab-bbab-0d65e164ed1b admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:56.814602 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227d1e54-bf3f-43ab-bbab-0d65e164ed1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:56.815083 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227d1e54-bf3f-43ab-bbab-0d65e164ed1b admin admin] PATH_INFO: `/v3/projects/68ec334ca1994b729290d6acef7e58ae/users/3344ced82dec4a5b950ac38a4c8c0d82/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:56.842809 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227d1e54-bf3f-43ab-bbab-0d65e164ed1b admin admin] RBAC: Authorizing `identity:create_grant(project_id=68ec334ca1994b729290d6acef7e58ae, user_id=3344ced82dec4a5b950ac38a4c8c0d82, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:56.846760 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227d1e54-bf3f-43ab-bbab-0d65e164ed1b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:56.868481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 889/1784] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:56 2022] PUT /identity/v3/projects/68ec334ca1994b729290d6acef7e58ae/users/3344ced82dec4a5b950ac38a4c8c0d82/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:56.869670 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227d1e54-bf3f-43ab-bbab-0d65e164ed1b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:56.917423 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95417ba3-2719-43b7-bdd0-0c436ba27c89 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:56.920542 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95417ba3-2719-43b7-bdd0-0c436ba27c89 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:56.936664 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 896/1785] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:56 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:56.939368 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74336f87-e5b3-4be1-87f0-9e48008b35dc None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:56.939661 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74336f87-e5b3-4be1-87f0-9e48008b35dc None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:56.939942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74336f87-e5b3-4be1-87f0-9e48008b35dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:56.979822 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-74336f87-e5b3-4be1-87f0-9e48008b35dc None None] MFA Rules not processed for user `3344ced82dec4a5b950ac38a4c8c0d82`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:56.981290 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-74336f87-e5b3-4be1-87f0-9e48008b35dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:57.004049 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227d1e54-bf3f-43ab-bbab-0d65e164ed1b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:57.015752 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba95f39-e724-489c-82c8-193c2a77443d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:57.017754 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba95f39-e724-489c-82c8-193c2a77443d admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:57.018183 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba95f39-e724-489c-82c8-193c2a77443d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:57.019782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba95f39-e724-489c-82c8-193c2a77443d admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:57.021762 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba95f39-e724-489c-82c8-193c2a77443d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:57.024728 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba95f39-e724-489c-82c8-193c2a77443d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:57.100098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 890/1786] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:56 2022] POST /identity/v3/projects => generated 355 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:57.101300 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba95f39-e724-489c-82c8-193c2a77443d admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:57.105402 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:57.121362 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 897/1787] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:56 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:57.127255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74336f87-e5b3-4be1-87f0-9e48008b35dc None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:57.140557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74336f87-e5b3-4be1-87f0-9e48008b35dc None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:57.152626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:57.270561 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba95f39-e724-489c-82c8-193c2a77443d admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:57.305361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97a0501-5831-4d19-8f28-ad69d6d5fe68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:57.308934 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97a0501-5831-4d19-8f28-ad69d6d5fe68 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:57.309322 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97a0501-5831-4d19-8f28-ad69d6d5fe68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:57.313845 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97a0501-5831-4d19-8f28-ad69d6d5fe68 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:57.314603 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97a0501-5831-4d19-8f28-ad69d6d5fe68 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:57.326184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97a0501-5831-4d19-8f28-ad69d6d5fe68 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:57.332173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fed7e3-a4d6-436f-ae5f-df8b01dbc02a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:57.333055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fed7e3-a4d6-436f-ae5f-df8b01dbc02a admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:57.333518 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fed7e3-a4d6-436f-ae5f-df8b01dbc02a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:57.333971 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fed7e3-a4d6-436f-ae5f-df8b01dbc02a admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:57.334782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fed7e3-a4d6-436f-ae5f-df8b01dbc02a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:57.343628 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fed7e3-a4d6-436f-ae5f-df8b01dbc02a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:57.368238 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 891/1788] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:57 2022] POST /identity/v3/users => generated 284 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:57.369385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7888ad-863b-4527-bb59-fd98f9842ab5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:57.369698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7888ad-863b-4527-bb59-fd98f9842ab5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:57.369975 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7888ad-863b-4527-bb59-fd98f9842ab5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:57.392091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 898/1789] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:18:57 2022] POST /identity/v3/projects => generated 407 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:18:57.393302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fed7e3-a4d6-436f-ae5f-df8b01dbc02a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:57.412082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9d7888ad-863b-4527-bb59-fd98f9842ab5 None None] MFA Rules not processed for user `e2846ad486314958bb74af71b647c470`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:57.413416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9d7888ad-863b-4527-bb59-fd98f9842ab5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:57.478602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fed7e3-a4d6-436f-ae5f-df8b01dbc02a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:57.485722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1affc62e-b392-445d-af5e-d1f2d195f0fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:57.486726 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1affc62e-b392-445d-af5e-d1f2d195f0fc admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:57.487197 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1affc62e-b392-445d-af5e-d1f2d195f0fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:57.487618 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1affc62e-b392-445d-af5e-d1f2d195f0fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:57.488418 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1affc62e-b392-445d-af5e-d1f2d195f0fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:57.490239 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1affc62e-b392-445d-af5e-d1f2d195f0fc admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:57.525857 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 899/1790] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:57 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:57.526964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1affc62e-b392-445d-af5e-d1f2d195f0fc admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:57.527684 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 892/1791] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:57 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:57.528823 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7888ad-863b-4527-bb59-fd98f9842ab5 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:57.539953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1affc62e-b392-445d-af5e-d1f2d195f0fc admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:57.541541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7888ad-863b-4527-bb59-fd98f9842ab5 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:57.545794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcebe616-52de-4ad1-9bbe-c64b36600ea3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:57.546456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcebe616-52de-4ad1-9bbe-c64b36600ea3 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:57.546737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcebe616-52de-4ad1-9bbe-c64b36600ea3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:57.547053 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcebe616-52de-4ad1-9bbe-c64b36600ea3 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:57.547681 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcebe616-52de-4ad1-9bbe-c64b36600ea3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:57.549602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcebe616-52de-4ad1-9bbe-c64b36600ea3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:57.552627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b047ff6-43b3-4c8b-8e98-32499a057f31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:57.553569 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b047ff6-43b3-4c8b-8e98-32499a057f31 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:57.553993 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b047ff6-43b3-4c8b-8e98-32499a057f31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:57.554443 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b047ff6-43b3-4c8b-8e98-32499a057f31 admin admin] PATH_INFO: `/v3/projects/bba09ef23dae4910a89daa82bb8f447a/users/56e941e0cd314f8fa67896e4801500ad/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:57.579902 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 900/1792] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:18:57 2022] POST /identity/v3/users => generated 310 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:18:57.584095 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b047ff6-43b3-4c8b-8e98-32499a057f31 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bba09ef23dae4910a89daa82bb8f447a, user_id=56e941e0cd314f8fa67896e4801500ad, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:57.599724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcebe616-52de-4ad1-9bbe-c64b36600ea3 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:57.602877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b047ff6-43b3-4c8b-8e98-32499a057f31 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:57.637160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcebe616-52de-4ad1-9bbe-c64b36600ea3 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:57.647893 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 893/1793] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:57 2022] PUT /identity/v3/projects/bba09ef23dae4910a89daa82bb8f447a/users/56e941e0cd314f8fa67896e4801500ad/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:57.652827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b047ff6-43b3-4c8b-8e98-32499a057f31 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:57.757522 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d90b801-0bf9-4c3c-8de6-266ea3af2305 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:57.759323 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d90b801-0bf9-4c3c-8de6-266ea3af2305 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:57.759656 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d90b801-0bf9-4c3c-8de6-266ea3af2305 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:57.759938 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d90b801-0bf9-4c3c-8de6-266ea3af2305 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:57.762407 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d90b801-0bf9-4c3c-8de6-266ea3af2305 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:57.764054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d90b801-0bf9-4c3c-8de6-266ea3af2305 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:57.783177 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:57.808248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 901/1794] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:57 2022] GET /identity/v3/roles => generated 1420 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:18:57.812517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d90b801-0bf9-4c3c-8de6-266ea3af2305 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:57.837512 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d90b801-0bf9-4c3c-8de6-266ea3af2305 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:57.851623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ff3c3c-7629-48bf-bf64-c418b0b33cfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:57.852308 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ff3c3c-7629-48bf-bf64-c418b0b33cfd admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:57.852638 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ff3c3c-7629-48bf-bf64-c418b0b33cfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:57.852906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ff3c3c-7629-48bf-bf64-c418b0b33cfd admin admin] PATH_INFO: `/v3/projects/027af2c11f494e59a4cbbd6158ed3b64/users/46bc96728c3144b7be8cf1947381b359/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:57.865678 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b047ff6-43b3-4c8b-8e98-32499a057f31 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:57.877397 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c9fb40-2802-4083-b84f-ec0d8253e5cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:57.878022 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c9fb40-2802-4083-b84f-ec0d8253e5cc admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:57.878297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c9fb40-2802-4083-b84f-ec0d8253e5cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:57.878564 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c9fb40-2802-4083-b84f-ec0d8253e5cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:57.879308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c9fb40-2802-4083-b84f-ec0d8253e5cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:57.880946 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c9fb40-2802-4083-b84f-ec0d8253e5cc admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:57.892434 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ff3c3c-7629-48bf-bf64-c418b0b33cfd admin admin] RBAC: Authorizing `identity:create_grant(project_id=027af2c11f494e59a4cbbd6158ed3b64, user_id=46bc96728c3144b7be8cf1947381b359, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:57.895766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ff3c3c-7629-48bf-bf64-c418b0b33cfd admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:57.902216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 894/1795] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:57 2022] GET /identity/v3/roles => generated 1420 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:57.903553 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c9fb40-2802-4083-b84f-ec0d8253e5cc admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:57.916378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c9fb40-2802-4083-b84f-ec0d8253e5cc admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:57.917198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 902/1796] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:57 2022] PUT /identity/v3/projects/027af2c11f494e59a4cbbd6158ed3b64/users/46bc96728c3144b7be8cf1947381b359/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:57.921971 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ff3c3c-7629-48bf-bf64-c418b0b33cfd admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:57.956922 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:57.998654 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ff3c3c-7629-48bf-bf64-c418b0b33cfd admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:58.007659 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28acbe4b-0e3b-4a2a-8b02-befa57fa5584 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:58.008554 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28acbe4b-0e3b-4a2a-8b02-befa57fa5584 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:58.008845 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28acbe4b-0e3b-4a2a-8b02-befa57fa5584 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:58.009151 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28acbe4b-0e3b-4a2a-8b02-befa57fa5584 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:58.009858 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28acbe4b-0e3b-4a2a-8b02-befa57fa5584 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:58.014117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28acbe4b-0e3b-4a2a-8b02-befa57fa5584 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:58.046599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 903/1797] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:18:57 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:18:58.051471 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28acbe4b-0e3b-4a2a-8b02-befa57fa5584 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:58.057827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4889e5-0dee-43c1-b2cb-53f191939d44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:58.058523 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4889e5-0dee-43c1-b2cb-53f191939d44 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:58.058982 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4889e5-0dee-43c1-b2cb-53f191939d44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:58.059256 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4889e5-0dee-43c1-b2cb-53f191939d44 admin admin] PATH_INFO: `/v3/projects/bba09ef23dae4910a89daa82bb8f447a/users/56e941e0cd314f8fa67896e4801500ad/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:58.061557 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4889e5-0dee-43c1-b2cb-53f191939d44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bba09ef23dae4910a89daa82bb8f447a, user_id=56e941e0cd314f8fa67896e4801500ad, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:58.064951 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4889e5-0dee-43c1-b2cb-53f191939d44 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:58.072966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28acbe4b-0e3b-4a2a-8b02-befa57fa5584 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:58.076921 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 895/1798] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:57 2022] PUT /identity/v3/projects/bba09ef23dae4910a89daa82bb8f447a/users/56e941e0cd314f8fa67896e4801500ad/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:58.091054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408e246b-653b-4bce-9b9d-93421da7f2a6 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:58.091351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408e246b-653b-4bce-9b9d-93421da7f2a6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:58.091609 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408e246b-653b-4bce-9b9d-93421da7f2a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:58.161382 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-408e246b-653b-4bce-9b9d-93421da7f2a6 None None] MFA Rules not processed for user `56e941e0cd314f8fa67896e4801500ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:58.162728 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-408e246b-653b-4bce-9b9d-93421da7f2a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:58.185309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252f75a6-df5d-46d2-8aa6-4dcc50bccb99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:58.185993 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252f75a6-df5d-46d2-8aa6-4dcc50bccb99 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:58.186526 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252f75a6-df5d-46d2-8aa6-4dcc50bccb99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:58.188958 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252f75a6-df5d-46d2-8aa6-4dcc50bccb99 admin admin] PATH_INFO: `/v3/projects/027af2c11f494e59a4cbbd6158ed3b64/users/46bc96728c3144b7be8cf1947381b359/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:58.192447 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-252f75a6-df5d-46d2-8aa6-4dcc50bccb99 admin admin] RBAC: Authorizing `identity:create_grant(project_id=027af2c11f494e59a4cbbd6158ed3b64, user_id=46bc96728c3144b7be8cf1947381b359, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:58.196470 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-252f75a6-df5d-46d2-8aa6-4dcc50bccb99 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:58.212154 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 904/1799] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:18:58 2022] PUT /identity/v3/projects/027af2c11f494e59a4cbbd6158ed3b64/users/46bc96728c3144b7be8cf1947381b359/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:18:58.226753 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa09b16-2438-4f36-9249-f35917361268 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:58.227193 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa09b16-2438-4f36-9249-f35917361268 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:58.227678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa09b16-2438-4f36-9249-f35917361268 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:58.267584 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-cfa09b16-2438-4f36-9249-f35917361268 None None] MFA Rules not processed for user `46bc96728c3144b7be8cf1947381b359`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:18:58.268965 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-cfa09b16-2438-4f36-9249-f35917361268 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:18:58.294448 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 896/1800] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:58 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:58.302625 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408e246b-653b-4bce-9b9d-93421da7f2a6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:58.334965 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:58.388927 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 905/1801] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:18:58 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:18:58.460468 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa09b16-2438-4f36-9249-f35917361268 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:58.482464 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408e246b-653b-4bce-9b9d-93421da7f2a6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:58.562990 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa09b16-2438-4f36-9249-f35917361268 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:58.568201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9d71ec-8c5f-4ae5-8d44-7dbe2c68e54c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:58.569353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9d71ec-8c5f-4ae5-8d44-7dbe2c68e54c service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:58.569941 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9d71ec-8c5f-4ae5-8d44-7dbe2c68e54c service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:58.570415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9d71ec-8c5f-4ae5-8d44-7dbe2c68e54c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:58.589739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9d71ec-8c5f-4ae5-8d44-7dbe2c68e54c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:58.592121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9d71ec-8c5f-4ae5-8d44-7dbe2c68e54c service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:58.608533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af1b787-8067-4349-9d19-aa0fb0c04ae4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:58.609984 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af1b787-8067-4349-9d19-aa0fb0c04ae4 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:58.610301 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af1b787-8067-4349-9d19-aa0fb0c04ae4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:58.611089 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af1b787-8067-4349-9d19-aa0fb0c04ae4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:58.618017 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 906/1802] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:58 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:18:58.632926 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:18:58.753888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af1b787-8067-4349-9d19-aa0fb0c04ae4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:58.769589 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af1b787-8067-4349-9d19-aa0fb0c04ae4 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:58.815192 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 897/1803] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:58 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 513 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:59.078607 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9d71ec-8c5f-4ae5-8d44-7dbe2c68e54c service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:59.092172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9d71ec-8c5f-4ae5-8d44-7dbe2c68e54c service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:59.117494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b4d911-12b5-40e6-aced-7347c5ba740f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:59.118914 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b4d911-12b5-40e6-aced-7347c5ba740f service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:59.119384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b4d911-12b5-40e6-aced-7347c5ba740f service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:59.119816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b4d911-12b5-40e6-aced-7347c5ba740f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:59.142009 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af1b787-8067-4349-9d19-aa0fb0c04ae4 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:18:59.153600 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af1b787-8067-4349-9d19-aa0fb0c04ae4 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:18:59.159062 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdbbfa9-3a31-4cfa-b3e8-8c5b1a3585f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:18:59.164002 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdbbfa9-3a31-4cfa-b3e8-8c5b1a3585f6 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:59.164568 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdbbfa9-3a31-4cfa-b3e8-8c5b1a3585f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:59.164568 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdbbfa9-3a31-4cfa-b3e8-8c5b1a3585f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:18:59.303271 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b4d911-12b5-40e6-aced-7347c5ba740f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:59.305444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b4d911-12b5-40e6-aced-7347c5ba740f service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:59.317412 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 907/1804] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:59 2022] GET /identity/v3/auth/tokens => generated 3282 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:18:59.356469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdbbfa9-3a31-4cfa-b3e8-8c5b1a3585f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:18:59.358973 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdbbfa9-3a31-4cfa-b3e8-8c5b1a3585f6 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:18:59.392206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 898/1805] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:18:59 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 252 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:18:59.810815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027497ac-893a-42c5-b7ef-aa17213f26cc None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:18:59.811615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027497ac-893a-42c5-b7ef-aa17213f26cc None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:18:59.812850 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027497ac-893a-42c5-b7ef-aa17213f26cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:00.171894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-027497ac-893a-42c5-b7ef-aa17213f26cc None None] MFA Rules not processed for user `e28a0ba45389417ba8ae92376dba87ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:00.175484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-027497ac-893a-42c5-b7ef-aa17213f26cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:00.399501 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 908/1806] 10.222.0.15 () {58 vars in 1009 bytes} [Tue Sep 13 01:18:59 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 589 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:00.620954 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdbbfa9-3a31-4cfa-b3e8-8c5b1a3585f6 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:00.622791 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0fdbbfa9-3a31-4cfa-b3e8-8c5b1a3585f6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:00.936921 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdbbfa9-3a31-4cfa-b3e8-8c5b1a3585f6 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:00.957480 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9919b3f-1064-440e-9627-b2e262404fc6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d7d8ed4ede4ddca192fce0a497689d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:00.960869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9919b3f-1064-440e-9627-b2e262404fc6 service swift] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:00.961172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9919b3f-1064-440e-9627-b2e262404fc6 service swift] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:00.961623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9919b3f-1064-440e-9627-b2e262404fc6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:00.980854 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9919b3f-1064-440e-9627-b2e262404fc6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:00.989085 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9919b3f-1064-440e-9627-b2e262404fc6 service swift] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:01.044413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 899/1807] 10.222.0.15 () {60 vars in 1384 bytes} [Tue Sep 13 01:19:00 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 01:19:02.297725 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027497ac-893a-42c5-b7ef-aa17213f26cc None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:02.303075 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:02.472635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027497ac-893a-42c5-b7ef-aa17213f26cc None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:02.478407 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03cedd0-affd-4017-aaff-6d025451e252 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:02.479311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03cedd0-affd-4017-aaff-6d025451e252 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:02.479594 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03cedd0-affd-4017-aaff-6d025451e252 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:02.479868 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03cedd0-affd-4017-aaff-6d025451e252 admin admin] PATH_INFO: `/v3/users/26198591c6774b4c87b5916420bbb47d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:02.481724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03cedd0-affd-4017-aaff-6d025451e252 admin admin] RBAC: Authorizing `identity:delete_user(user_id=26198591c6774b4c87b5916420bbb47d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:02.483741 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03cedd0-affd-4017-aaff-6d025451e252 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:02.560736 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c03cedd0-affd-4017-aaff-6d025451e252 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26198591c6774b4c87b5916420bbb47d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:02.562832 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c03cedd0-affd-4017-aaff-6d025451e252 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26198591c6774b4c87b5916420bbb47d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:02.574452 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c03cedd0-affd-4017-aaff-6d025451e252 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26198591c6774b4c87b5916420bbb47d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:02.575520 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c03cedd0-affd-4017-aaff-6d025451e252 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26198591c6774b4c87b5916420bbb47d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:02.597308 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c03cedd0-affd-4017-aaff-6d025451e252 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26198591c6774b4c87b5916420bbb47d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:02.619424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 909/1808] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:02 2022] DELETE /identity/v3/users/26198591c6774b4c87b5916420bbb47d => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:02.623847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00ab19e-7231-447c-a8be-24fcb1eb1c39 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:02.624943 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00ab19e-7231-447c-a8be-24fcb1eb1c39 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:02.625399 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00ab19e-7231-447c-a8be-24fcb1eb1c39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:02.647824 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c00ab19e-7231-447c-a8be-24fcb1eb1c39 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:02.649428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c00ab19e-7231-447c-a8be-24fcb1eb1c39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:02.664171 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:02.742773 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 900/1809] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:02.752627 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03cedd0-affd-4017-aaff-6d025451e252 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:02.754433 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c03cedd0-affd-4017-aaff-6d025451e252 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:02.877831 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03cedd0-affd-4017-aaff-6d025451e252 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:02.882246 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d91a9f8-f430-45d9-9657-5b54be7680a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:02.882997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d91a9f8-f430-45d9-9657-5b54be7680a0 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:02.883284 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d91a9f8-f430-45d9-9657-5b54be7680a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:02.883639 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d91a9f8-f430-45d9-9657-5b54be7680a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:02.898963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d91a9f8-f430-45d9-9657-5b54be7680a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:02.904753 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d91a9f8-f430-45d9-9657-5b54be7680a0 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:02.989506 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 910/1810] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:03.435986 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00ab19e-7231-447c-a8be-24fcb1eb1c39 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:03.444230 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c00ab19e-7231-447c-a8be-24fcb1eb1c39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:03.467187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00ab19e-7231-447c-a8be-24fcb1eb1c39 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:03.472853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565c3c62-6e94-4393-96c7-8dccb4250531 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:03.473503 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565c3c62-6e94-4393-96c7-8dccb4250531 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:03.473782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565c3c62-6e94-4393-96c7-8dccb4250531 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:03.474051 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565c3c62-6e94-4393-96c7-8dccb4250531 admin admin] PATH_INFO: `/v3/projects/b3d92f0132c84f26b55b2a75a4328955` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:03.475648 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565c3c62-6e94-4393-96c7-8dccb4250531 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b3d92f0132c84f26b55b2a75a4328955)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:03.477570 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565c3c62-6e94-4393-96c7-8dccb4250531 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:03.580227 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-565c3c62-6e94-4393-96c7-8dccb4250531 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b3d92f0132c84f26b55b2a75a4328955', 'request_id': 'req-565c3c62-6e94-4393-96c7-8dccb4250531'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:03.588123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-565c3c62-6e94-4393-96c7-8dccb4250531 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b3d92f0132c84f26b55b2a75a4328955', 'request_id': 'req-565c3c62-6e94-4393-96c7-8dccb4250531'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:03.595974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-565c3c62-6e94-4393-96c7-8dccb4250531 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b3d92f0132c84f26b55b2a75a4328955', 'request_id': 'req-565c3c62-6e94-4393-96c7-8dccb4250531'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:03.612327 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-565c3c62-6e94-4393-96c7-8dccb4250531 admin admin] The token cache is being invalidate because project b3d92f0132c84f26b55b2a75a4328955 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:03.612605 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-565c3c62-6e94-4393-96c7-8dccb4250531 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:03.613309 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-565c3c62-6e94-4393-96c7-8dccb4250531 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:03.614947 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 901/1811] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:03 2022] DELETE /identity/v3/projects/b3d92f0132c84f26b55b2a75a4328955 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:03.619186 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d91a9f8-f430-45d9-9657-5b54be7680a0 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:03.628920 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d91a9f8-f430-45d9-9657-5b54be7680a0 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:03.649037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:03.687545 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02873c9-763f-4ace-b8ac-23d2f76ebe79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:03.688174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02873c9-763f-4ace-b8ac-23d2f76ebe79 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:03.688454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02873c9-763f-4ace-b8ac-23d2f76ebe79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:03.688834 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02873c9-763f-4ace-b8ac-23d2f76ebe79 admin admin] PATH_INFO: `/v3/users/8852beeb168442e4b99996e2a8189937` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:03.689877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02873c9-763f-4ace-b8ac-23d2f76ebe79 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8852beeb168442e4b99996e2a8189937)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:03.691821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02873c9-763f-4ace-b8ac-23d2f76ebe79 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:03.740117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f02873c9-763f-4ace-b8ac-23d2f76ebe79 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8852beeb168442e4b99996e2a8189937'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:03.740764 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f02873c9-763f-4ace-b8ac-23d2f76ebe79 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8852beeb168442e4b99996e2a8189937'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:03.748454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f02873c9-763f-4ace-b8ac-23d2f76ebe79 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8852beeb168442e4b99996e2a8189937'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:03.749114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f02873c9-763f-4ace-b8ac-23d2f76ebe79 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8852beeb168442e4b99996e2a8189937'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:03.762103 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f02873c9-763f-4ace-b8ac-23d2f76ebe79 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8852beeb168442e4b99996e2a8189937'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:03.777925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 911/1812] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:03 2022] DELETE /identity/v3/users/8852beeb168442e4b99996e2a8189937 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:04.109989 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565c3c62-6e94-4393-96c7-8dccb4250531 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:04.112136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-565c3c62-6e94-4393-96c7-8dccb4250531 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:04.131258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565c3c62-6e94-4393-96c7-8dccb4250531 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:04.192271 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c938228-3f20-495b-8113-94d77eae2ccf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:04.193023 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c938228-3f20-495b-8113-94d77eae2ccf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:04.193297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c938228-3f20-495b-8113-94d77eae2ccf admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:04.193565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c938228-3f20-495b-8113-94d77eae2ccf admin admin] PATH_INFO: `/v3/projects/9b4cfdf1aede49af91f83f67637e4722` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:04.194716 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c938228-3f20-495b-8113-94d77eae2ccf admin admin] RBAC: Authorizing `identity:delete_project(project_id=9b4cfdf1aede49af91f83f67637e4722)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:04.196657 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c938228-3f20-495b-8113-94d77eae2ccf admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:04.245377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:04.250271 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4c938228-3f20-495b-8113-94d77eae2ccf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9b4cfdf1aede49af91f83f67637e4722', 'request_id': 'req-4c938228-3f20-495b-8113-94d77eae2ccf'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:04.258617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4c938228-3f20-495b-8113-94d77eae2ccf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9b4cfdf1aede49af91f83f67637e4722', 'request_id': 'req-4c938228-3f20-495b-8113-94d77eae2ccf'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:04.266987 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4c938228-3f20-495b-8113-94d77eae2ccf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9b4cfdf1aede49af91f83f67637e4722', 'request_id': 'req-4c938228-3f20-495b-8113-94d77eae2ccf'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:04.285960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4c938228-3f20-495b-8113-94d77eae2ccf admin admin] The token cache is being invalidate because project 9b4cfdf1aede49af91f83f67637e4722 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:04.286293 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4c938228-3f20-495b-8113-94d77eae2ccf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:04.287198 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4c938228-3f20-495b-8113-94d77eae2ccf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:04.292435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 902/1813] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:04 2022] DELETE /identity/v3/projects/9b4cfdf1aede49af91f83f67637e4722 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:04.294076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f2fb74-683c-4b05-8a7a-e6a2aa82ff59 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:04.294364 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f2fb74-683c-4b05-8a7a-e6a2aa82ff59 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:04.294644 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f2fb74-683c-4b05-8a7a-e6a2aa82ff59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:04.326152 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-76f2fb74-683c-4b05-8a7a-e6a2aa82ff59 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:04.327657 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-76f2fb74-683c-4b05-8a7a-e6a2aa82ff59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:04.440358 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 912/1814] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:04.447700 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af0103b-15f8-4501-b4d7-058cc127d6f2 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:04.447988 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af0103b-15f8-4501-b4d7-058cc127d6f2 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:04.448265 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af0103b-15f8-4501-b4d7-058cc127d6f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:04.470777 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-1af0103b-15f8-4501-b4d7-058cc127d6f2 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:04.472154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1af0103b-15f8-4501-b4d7-058cc127d6f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:04.481223 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 903/1815] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:04.485500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f2fb74-683c-4b05-8a7a-e6a2aa82ff59 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:04.496152 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f2fb74-683c-4b05-8a7a-e6a2aa82ff59 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:04.502010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bf2e9b-d029-4113-bdd3-3e6fcaf8aa5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:04.502619 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bf2e9b-d029-4113-bdd3-3e6fcaf8aa5c admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:04.502933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bf2e9b-d029-4113-bdd3-3e6fcaf8aa5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:04.503212 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bf2e9b-d029-4113-bdd3-3e6fcaf8aa5c admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:04.503873 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bf2e9b-d029-4113-bdd3-3e6fcaf8aa5c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:19:04.504185 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bf2e9b-d029-4113-bdd3-3e6fcaf8aa5c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:04.505777 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bf2e9b-d029-4113-bdd3-3e6fcaf8aa5c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:04.518958 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 913/1816] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:19:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:19:04.523423 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af0103b-15f8-4501-b4d7-058cc127d6f2 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:04.533050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af0103b-15f8-4501-b4d7-058cc127d6f2 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:04.537773 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40896673-9237-4982-8079-78a254ea346b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:04.538405 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40896673-9237-4982-8079-78a254ea346b admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:04.538687 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40896673-9237-4982-8079-78a254ea346b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:04.539001 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40896673-9237-4982-8079-78a254ea346b admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:04.539637 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40896673-9237-4982-8079-78a254ea346b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:04.541541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40896673-9237-4982-8079-78a254ea346b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:04.573765 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 904/1817] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:04 2022] POST /identity/v3/projects => generated 387 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:04.577904 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bf2e9b-d029-4113-bdd3-3e6fcaf8aa5c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:04.589104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bf2e9b-d029-4113-bdd3-3e6fcaf8aa5c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:04.621037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:04.656930 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37c9530-fa4a-4f25-8c8a-c60cd57ce3e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:04.657552 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37c9530-fa4a-4f25-8c8a-c60cd57ce3e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:04.657854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37c9530-fa4a-4f25-8c8a-c60cd57ce3e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:04.658110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37c9530-fa4a-4f25-8c8a-c60cd57ce3e1 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:04.658749 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37c9530-fa4a-4f25-8c8a-c60cd57ce3e1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:04.660761 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37c9530-fa4a-4f25-8c8a-c60cd57ce3e1 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:04.693638 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 914/1818] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:04 2022] POST /identity/v3/users => generated 294 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:19:04.699793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40896673-9237-4982-8079-78a254ea346b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:04.713033 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40896673-9237-4982-8079-78a254ea346b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:04.720925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613aa72c-86ff-43a5-a915-b2fec276cdbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:04.726380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613aa72c-86ff-43a5-a915-b2fec276cdbb admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:04.726982 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613aa72c-86ff-43a5-a915-b2fec276cdbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:04.727491 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613aa72c-86ff-43a5-a915-b2fec276cdbb admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:04.728363 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613aa72c-86ff-43a5-a915-b2fec276cdbb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:04.730675 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613aa72c-86ff-43a5-a915-b2fec276cdbb admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:04.742886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 905/1819] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:04 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:04.747323 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37c9530-fa4a-4f25-8c8a-c60cd57ce3e1 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:04.764393 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37c9530-fa4a-4f25-8c8a-c60cd57ce3e1 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:04.771745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acbed4ed-bf2f-4584-acad-fba79a5b9ff1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:04.772398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbed4ed-bf2f-4584-acad-fba79a5b9ff1 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:04.773323 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbed4ed-bf2f-4584-acad-fba79a5b9ff1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:04.773448 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbed4ed-bf2f-4584-acad-fba79a5b9ff1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:04.774259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acbed4ed-bf2f-4584-acad-fba79a5b9ff1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:04.775968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acbed4ed-bf2f-4584-acad-fba79a5b9ff1 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:04.794710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 915/1820] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:04 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:19:04.796830 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613aa72c-86ff-43a5-a915-b2fec276cdbb admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:04.810740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613aa72c-86ff-43a5-a915-b2fec276cdbb admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:04.824402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d4f268-afed-4c28-a23e-7040dbcbb50f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:04.825282 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d4f268-afed-4c28-a23e-7040dbcbb50f admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:04.825641 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d4f268-afed-4c28-a23e-7040dbcbb50f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:04.825994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d4f268-afed-4c28-a23e-7040dbcbb50f admin admin] PATH_INFO: `/v3/projects/825558f7e8c240b899bdbdcb1e34d377/users/f0c579c7a8bc4bf2a04298763ca2621f/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:04.864510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d4f268-afed-4c28-a23e-7040dbcbb50f admin admin] RBAC: Authorizing `identity:create_grant(project_id=825558f7e8c240b899bdbdcb1e34d377, user_id=f0c579c7a8bc4bf2a04298763ca2621f, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:04.870728 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d4f268-afed-4c28-a23e-7040dbcbb50f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:04.900440 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 906/1821] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:04 2022] PUT /identity/v3/projects/825558f7e8c240b899bdbdcb1e34d377/users/f0c579c7a8bc4bf2a04298763ca2621f/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:04.914079 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866d3c04-4b69-40d5-afd1-45bcbc794461 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:04.914371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866d3c04-4b69-40d5-afd1-45bcbc794461 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:04.914650 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866d3c04-4b69-40d5-afd1-45bcbc794461 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:04.955967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-866d3c04-4b69-40d5-afd1-45bcbc794461 None None] MFA Rules not processed for user `f0c579c7a8bc4bf2a04298763ca2621f`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:04.957962 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-866d3c04-4b69-40d5-afd1-45bcbc794461 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:05.102732 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 916/1822] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:04 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:05.107078 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d4f268-afed-4c28-a23e-7040dbcbb50f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:05.217011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d4f268-afed-4c28-a23e-7040dbcbb50f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:05.222721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce7e5d0-51ba-4f01-9999-17291f14091d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:05.223387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce7e5d0-51ba-4f01-9999-17291f14091d admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:05.223667 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce7e5d0-51ba-4f01-9999-17291f14091d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:05.223943 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce7e5d0-51ba-4f01-9999-17291f14091d admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:05.224579 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce7e5d0-51ba-4f01-9999-17291f14091d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:05.226592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce7e5d0-51ba-4f01-9999-17291f14091d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:05.260076 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 907/1823] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:05 2022] POST /identity/v3/projects => generated 387 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:05.264009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866d3c04-4b69-40d5-afd1-45bcbc794461 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:05.278294 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866d3c04-4b69-40d5-afd1-45bcbc794461 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:05.354420 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36597f70-4b15-4249-a837-ad9a5b7669a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:05.355152 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36597f70-4b15-4249-a837-ad9a5b7669a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:05.355436 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36597f70-4b15-4249-a837-ad9a5b7669a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:05.355754 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36597f70-4b15-4249-a837-ad9a5b7669a2 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:05.356514 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36597f70-4b15-4249-a837-ad9a5b7669a2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:05.359452 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36597f70-4b15-4249-a837-ad9a5b7669a2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:05.384803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 917/1824] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:05 2022] POST /identity/v3/users => generated 300 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:05.388619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce7e5d0-51ba-4f01-9999-17291f14091d admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:05.407893 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce7e5d0-51ba-4f01-9999-17291f14091d admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:05.421778 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32de556-d548-4b3a-a91b-c2cc38084e5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:05.422577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32de556-d548-4b3a-a91b-c2cc38084e5b admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:05.423014 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32de556-d548-4b3a-a91b-c2cc38084e5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:05.423362 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32de556-d548-4b3a-a91b-c2cc38084e5b admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:05.424147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32de556-d548-4b3a-a91b-c2cc38084e5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:05.426706 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32de556-d548-4b3a-a91b-c2cc38084e5b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:05.460233 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 908/1825] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:05 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:05.461401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36597f70-4b15-4249-a837-ad9a5b7669a2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:05.483189 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36597f70-4b15-4249-a837-ad9a5b7669a2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:05.489236 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ecb49a-c997-47a6-ae66-726867ca68ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:05.490188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ecb49a-c997-47a6-ae66-726867ca68ac admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:05.490565 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ecb49a-c997-47a6-ae66-726867ca68ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:05.491088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ecb49a-c997-47a6-ae66-726867ca68ac admin admin] PATH_INFO: `/v3/projects/c30a8c79fc0244b1a72954cd4280c096/users/26e7d2d2c5494416a21e06295e806bea/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:05.533819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ecb49a-c997-47a6-ae66-726867ca68ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=c30a8c79fc0244b1a72954cd4280c096, user_id=26e7d2d2c5494416a21e06295e806bea, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:05.540738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ecb49a-c997-47a6-ae66-726867ca68ac admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:05.584612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 918/1826] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:05 2022] PUT /identity/v3/projects/c30a8c79fc0244b1a72954cd4280c096/users/26e7d2d2c5494416a21e06295e806bea/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:05.588165 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32de556-d548-4b3a-a91b-c2cc38084e5b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:05.607285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32de556-d548-4b3a-a91b-c2cc38084e5b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:05.709993 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:05.751862 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba23d3dd-3366-4889-841a-db4386e52b68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:05.752490 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba23d3dd-3366-4889-841a-db4386e52b68 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:05.752774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba23d3dd-3366-4889-841a-db4386e52b68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:05.753048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba23d3dd-3366-4889-841a-db4386e52b68 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:05.753698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba23d3dd-3366-4889-841a-db4386e52b68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:05.755463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba23d3dd-3366-4889-841a-db4386e52b68 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:05.774254 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 909/1827] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:05 2022] GET /identity/v3/roles => generated 1420 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:05.779283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ecb49a-c997-47a6-ae66-726867ca68ac admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:05.807668 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:05.830460 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ecb49a-c997-47a6-ae66-726867ca68ac admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:05.836039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323df22c-f837-4d80-bea1-cf8309953cb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:05.836809 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323df22c-f837-4d80-bea1-cf8309953cb1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:05.837083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323df22c-f837-4d80-bea1-cf8309953cb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:05.837356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323df22c-f837-4d80-bea1-cf8309953cb1 admin admin] PATH_INFO: `/v3/projects/c30a8c79fc0244b1a72954cd4280c096/users/26e7d2d2c5494416a21e06295e806bea/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:05.839578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323df22c-f837-4d80-bea1-cf8309953cb1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c30a8c79fc0244b1a72954cd4280c096, user_id=26e7d2d2c5494416a21e06295e806bea, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:05.843665 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323df22c-f837-4d80-bea1-cf8309953cb1 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:05.874083 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 919/1828] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:05 2022] PUT /identity/v3/projects/c30a8c79fc0244b1a72954cd4280c096/users/26e7d2d2c5494416a21e06295e806bea/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:05.883920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b31b19-a54b-4d28-8408-dc7a8f35cae6 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:05.884206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b31b19-a54b-4d28-8408-dc7a8f35cae6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:05.884534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b31b19-a54b-4d28-8408-dc7a8f35cae6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:05.924336 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-77b31b19-a54b-4d28-8408-dc7a8f35cae6 None None] MFA Rules not processed for user `26e7d2d2c5494416a21e06295e806bea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:05.925426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-77b31b19-a54b-4d28-8408-dc7a8f35cae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:06.067795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 910/1829] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:05 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:06.069248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323df22c-f837-4d80-bea1-cf8309953cb1 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:06.071125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-323df22c-f837-4d80-bea1-cf8309953cb1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:06.156082 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323df22c-f837-4d80-bea1-cf8309953cb1 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:06.160864 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fe0c12-6e51-4793-b34d-7579d64c8d1b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:06.161560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fe0c12-6e51-4793-b34d-7579d64c8d1b service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:06.161990 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fe0c12-6e51-4793-b34d-7579d64c8d1b service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:06.162328 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fe0c12-6e51-4793-b34d-7579d64c8d1b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:06.178650 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fe0c12-6e51-4793-b34d-7579d64c8d1b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:06.180797 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fe0c12-6e51-4793-b34d-7579d64c8d1b service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:06.197434 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 920/1830] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:06 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:06.949894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b31b19-a54b-4d28-8408-dc7a8f35cae6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:07.029997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b31b19-a54b-4d28-8408-dc7a8f35cae6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:07.035484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a7e624-4aee-44c0-88b2-558810b44d7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:07.036235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a7e624-4aee-44c0-88b2-558810b44d7b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:07.036722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a7e624-4aee-44c0-88b2-558810b44d7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:07.037070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a7e624-4aee-44c0-88b2-558810b44d7b admin admin] PATH_INFO: `/v3/users/f0c579c7a8bc4bf2a04298763ca2621f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:07.039177 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a7e624-4aee-44c0-88b2-558810b44d7b admin admin] RBAC: Authorizing `identity:delete_user(user_id=f0c579c7a8bc4bf2a04298763ca2621f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:07.041339 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a7e624-4aee-44c0-88b2-558810b44d7b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:07.102742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d9a7e624-4aee-44c0-88b2-558810b44d7b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f0c579c7a8bc4bf2a04298763ca2621f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:07.103506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d9a7e624-4aee-44c0-88b2-558810b44d7b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f0c579c7a8bc4bf2a04298763ca2621f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:07.117852 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d9a7e624-4aee-44c0-88b2-558810b44d7b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f0c579c7a8bc4bf2a04298763ca2621f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:07.118540 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d9a7e624-4aee-44c0-88b2-558810b44d7b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f0c579c7a8bc4bf2a04298763ca2621f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:07.126617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d9a7e624-4aee-44c0-88b2-558810b44d7b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f0c579c7a8bc4bf2a04298763ca2621f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:07.128713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:07.147123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 911/1831] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:06 2022] DELETE /identity/v3/users/f0c579c7a8bc4bf2a04298763ca2621f => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:07.151703 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfd8a78-7553-4a4f-870e-c440b138e446 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:07.152189 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfd8a78-7553-4a4f-870e-c440b138e446 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:07.152262 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfd8a78-7553-4a4f-870e-c440b138e446 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:07.172613 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8dfd8a78-7553-4a4f-870e-c440b138e446 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:07.173828 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8dfd8a78-7553-4a4f-870e-c440b138e446 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:07.243873 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 921/1832] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:07.256602 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a7e624-4aee-44c0-88b2-558810b44d7b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:07.258076 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d9a7e624-4aee-44c0-88b2-558810b44d7b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:07.352386 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a7e624-4aee-44c0-88b2-558810b44d7b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:07.369177 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d5a31f-665b-4179-9ea0-64232b791487 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:07.370142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d5a31f-665b-4179-9ea0-64232b791487 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:07.372247 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d5a31f-665b-4179-9ea0-64232b791487 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:07.375299 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d5a31f-665b-4179-9ea0-64232b791487 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:07.420173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d5a31f-665b-4179-9ea0-64232b791487 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:07.422339 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d5a31f-665b-4179-9ea0-64232b791487 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:07.436976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 912/1833] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:07.845065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfd8a78-7553-4a4f-870e-c440b138e446 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:07.852232 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8dfd8a78-7553-4a4f-870e-c440b138e446 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:07.878328 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfd8a78-7553-4a4f-870e-c440b138e446 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:07.886834 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28927a75-335b-4eb5-9a5c-71a3d38e16e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:07.887807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28927a75-335b-4eb5-9a5c-71a3d38e16e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:07.887807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28927a75-335b-4eb5-9a5c-71a3d38e16e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:07.888375 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28927a75-335b-4eb5-9a5c-71a3d38e16e2 admin admin] PATH_INFO: `/v3/projects/825558f7e8c240b899bdbdcb1e34d377` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:07.889676 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28927a75-335b-4eb5-9a5c-71a3d38e16e2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=825558f7e8c240b899bdbdcb1e34d377)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:07.892029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28927a75-335b-4eb5-9a5c-71a3d38e16e2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:07.953812 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-28927a75-335b-4eb5-9a5c-71a3d38e16e2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '825558f7e8c240b899bdbdcb1e34d377', 'request_id': 'req-28927a75-335b-4eb5-9a5c-71a3d38e16e2'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:07.960234 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-28927a75-335b-4eb5-9a5c-71a3d38e16e2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '825558f7e8c240b899bdbdcb1e34d377', 'request_id': 'req-28927a75-335b-4eb5-9a5c-71a3d38e16e2'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:07.967157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-28927a75-335b-4eb5-9a5c-71a3d38e16e2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '825558f7e8c240b899bdbdcb1e34d377', 'request_id': 'req-28927a75-335b-4eb5-9a5c-71a3d38e16e2'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:07.983308 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-28927a75-335b-4eb5-9a5c-71a3d38e16e2 admin admin] The token cache is being invalidate because project 825558f7e8c240b899bdbdcb1e34d377 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:07.983611 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-28927a75-335b-4eb5-9a5c-71a3d38e16e2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:07.984398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-28927a75-335b-4eb5-9a5c-71a3d38e16e2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:07.986194 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 922/1834] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:07 2022] DELETE /identity/v3/projects/825558f7e8c240b899bdbdcb1e34d377 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:07.990407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d5a31f-665b-4179-9ea0-64232b791487 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:08.002426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d5a31f-665b-4179-9ea0-64232b791487 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:08.066004 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fda253b-74bd-4d85-b2dc-c36d80df8e0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:08.066654 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fda253b-74bd-4d85-b2dc-c36d80df8e0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:08.066963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fda253b-74bd-4d85-b2dc-c36d80df8e0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:08.067237 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fda253b-74bd-4d85-b2dc-c36d80df8e0c admin admin] PATH_INFO: `/v3/users/26e7d2d2c5494416a21e06295e806bea` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:08.068421 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fda253b-74bd-4d85-b2dc-c36d80df8e0c admin admin] RBAC: Authorizing `identity:delete_user(user_id=26e7d2d2c5494416a21e06295e806bea)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:08.070348 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fda253b-74bd-4d85-b2dc-c36d80df8e0c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:08.165488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5fda253b-74bd-4d85-b2dc-c36d80df8e0c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26e7d2d2c5494416a21e06295e806bea'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:08.167204 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5fda253b-74bd-4d85-b2dc-c36d80df8e0c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26e7d2d2c5494416a21e06295e806bea'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:08.184111 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5fda253b-74bd-4d85-b2dc-c36d80df8e0c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26e7d2d2c5494416a21e06295e806bea'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:08.184864 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5fda253b-74bd-4d85-b2dc-c36d80df8e0c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26e7d2d2c5494416a21e06295e806bea'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:08.198797 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5fda253b-74bd-4d85-b2dc-c36d80df8e0c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26e7d2d2c5494416a21e06295e806bea'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:08.242687 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 913/1835] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:07 2022] DELETE /identity/v3/users/26e7d2d2c5494416a21e06295e806bea => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:19:08.825567 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28927a75-335b-4eb5-9a5c-71a3d38e16e2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:08.828774 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-28927a75-335b-4eb5-9a5c-71a3d38e16e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:08.873426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28927a75-335b-4eb5-9a5c-71a3d38e16e2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:08.893031 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:08.986198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109adc8c-bbf4-4b9f-b542-adb47ecce0a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:08.986886 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109adc8c-bbf4-4b9f-b542-adb47ecce0a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:08.987171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109adc8c-bbf4-4b9f-b542-adb47ecce0a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:08.987448 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109adc8c-bbf4-4b9f-b542-adb47ecce0a6 admin admin] PATH_INFO: `/v3/projects/c30a8c79fc0244b1a72954cd4280c096` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:08.991931 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109adc8c-bbf4-4b9f-b542-adb47ecce0a6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c30a8c79fc0244b1a72954cd4280c096)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:08.994420 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109adc8c-bbf4-4b9f-b542-adb47ecce0a6 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:09.078561 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-109adc8c-bbf4-4b9f-b542-adb47ecce0a6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c30a8c79fc0244b1a72954cd4280c096', 'request_id': 'req-109adc8c-bbf4-4b9f-b542-adb47ecce0a6'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:09.085813 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-109adc8c-bbf4-4b9f-b542-adb47ecce0a6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c30a8c79fc0244b1a72954cd4280c096', 'request_id': 'req-109adc8c-bbf4-4b9f-b542-adb47ecce0a6'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:09.105458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-109adc8c-bbf4-4b9f-b542-adb47ecce0a6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c30a8c79fc0244b1a72954cd4280c096', 'request_id': 'req-109adc8c-bbf4-4b9f-b542-adb47ecce0a6'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:09.125983 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-109adc8c-bbf4-4b9f-b542-adb47ecce0a6 admin admin] The token cache is being invalidate because project c30a8c79fc0244b1a72954cd4280c096 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:09.126354 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-109adc8c-bbf4-4b9f-b542-adb47ecce0a6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:09.127054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-109adc8c-bbf4-4b9f-b542-adb47ecce0a6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:09.129244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 923/1836] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:08 2022] DELETE /identity/v3/projects/c30a8c79fc0244b1a72954cd4280c096 => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:09.134560 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da27bad5-4931-4f9b-b0c5-ee11a735365f None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:09.135240 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da27bad5-4931-4f9b-b0c5-ee11a735365f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:09.135240 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da27bad5-4931-4f9b-b0c5-ee11a735365f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:09.155661 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:09.182244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-da27bad5-4931-4f9b-b0c5-ee11a735365f None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:09.183584 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-da27bad5-4931-4f9b-b0c5-ee11a735365f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:09.304470 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 914/1837] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:09.315963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e830fd9-e3f2-408b-ad60-c6f3c9e27d3e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:09.315963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e830fd9-e3f2-408b-ad60-c6f3c9e27d3e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:09.316111 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e830fd9-e3f2-408b-ad60-c6f3c9e27d3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:09.354572 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-6e830fd9-e3f2-408b-ad60-c6f3c9e27d3e None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:09.356230 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6e830fd9-e3f2-408b-ad60-c6f3c9e27d3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:09.375071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da27bad5-4931-4f9b-b0c5-ee11a735365f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:09.375790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 924/1838] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:09.409229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da27bad5-4931-4f9b-b0c5-ee11a735365f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:09.427576 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2327d9-42a5-418b-b47f-0f1e80ceea07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:09.428349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2327d9-42a5-418b-b47f-0f1e80ceea07 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:09.428644 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2327d9-42a5-418b-b47f-0f1e80ceea07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:09.428914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2327d9-42a5-418b-b47f-0f1e80ceea07 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:09.429573 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2327d9-42a5-418b-b47f-0f1e80ceea07 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:19:09.429878 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2327d9-42a5-418b-b47f-0f1e80ceea07 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:09.431543 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2327d9-42a5-418b-b47f-0f1e80ceea07 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:09.466722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 915/1839] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:19:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:19:09.471985 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e830fd9-e3f2-408b-ad60-c6f3c9e27d3e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:09.486308 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e830fd9-e3f2-408b-ad60-c6f3c9e27d3e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:09.501755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a69ad95-0cf5-42d4-b470-5d2ae4253452 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:09.502705 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a69ad95-0cf5-42d4-b470-5d2ae4253452 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:09.505757 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a69ad95-0cf5-42d4-b470-5d2ae4253452 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:09.521216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a69ad95-0cf5-42d4-b470-5d2ae4253452 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:09.523815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a69ad95-0cf5-42d4-b470-5d2ae4253452 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:09.526189 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a69ad95-0cf5-42d4-b470-5d2ae4253452 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:09.634981 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 925/1840] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:09 2022] POST /identity/v3/projects => generated 385 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:09.635298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2327d9-42a5-418b-b47f-0f1e80ceea07 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:09.659484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2327d9-42a5-418b-b47f-0f1e80ceea07 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:09.724731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdadc7c-ef7c-4a87-a20a-1af8b1c9f561 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:09.725635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdadc7c-ef7c-4a87-a20a-1af8b1c9f561 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:09.726586 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdadc7c-ef7c-4a87-a20a-1af8b1c9f561 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:09.726978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdadc7c-ef7c-4a87-a20a-1af8b1c9f561 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:09.727815 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbdadc7c-ef7c-4a87-a20a-1af8b1c9f561 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:09.737284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbdadc7c-ef7c-4a87-a20a-1af8b1c9f561 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:09.760037 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 916/1841] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:09 2022] POST /identity/v3/users => generated 293 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:19:09.765220 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a69ad95-0cf5-42d4-b470-5d2ae4253452 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:09.776494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a69ad95-0cf5-42d4-b470-5d2ae4253452 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:09.781306 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b5a159-fb6b-496d-8542-acda1bac9714 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:09.781948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b5a159-fb6b-496d-8542-acda1bac9714 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:09.782256 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b5a159-fb6b-496d-8542-acda1bac9714 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:09.782488 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b5a159-fb6b-496d-8542-acda1bac9714 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:09.783228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b5a159-fb6b-496d-8542-acda1bac9714 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:09.784796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b5a159-fb6b-496d-8542-acda1bac9714 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:09.797880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 926/1842] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:09 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:19:09.802052 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdadc7c-ef7c-4a87-a20a-1af8b1c9f561 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:09.813790 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdadc7c-ef7c-4a87-a20a-1af8b1c9f561 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:09.819205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7945799c-f412-4a70-9ab5-c7cccc5e71ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:09.819782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7945799c-f412-4a70-9ab5-c7cccc5e71ff admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:09.819982 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7945799c-f412-4a70-9ab5-c7cccc5e71ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:09.820270 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7945799c-f412-4a70-9ab5-c7cccc5e71ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:09.820964 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7945799c-f412-4a70-9ab5-c7cccc5e71ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:09.822692 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7945799c-f412-4a70-9ab5-c7cccc5e71ff admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:09.837739 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 917/1843] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:09 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:19:09.839070 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b5a159-fb6b-496d-8542-acda1bac9714 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:09.852695 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b5a159-fb6b-496d-8542-acda1bac9714 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:09.857820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458182f1-a263-4514-9376-312e1d979ddb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:09.859415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458182f1-a263-4514-9376-312e1d979ddb admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:09.859897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458182f1-a263-4514-9376-312e1d979ddb admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:09.860365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458182f1-a263-4514-9376-312e1d979ddb admin admin] PATH_INFO: `/v3/projects/f0c6baba8aa248fdbaa02452ef21e987/users/a4dca1b96a7541698845aa6c79c71367/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:09.890466 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458182f1-a263-4514-9376-312e1d979ddb admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0c6baba8aa248fdbaa02452ef21e987, user_id=a4dca1b96a7541698845aa6c79c71367, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:09.893579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458182f1-a263-4514-9376-312e1d979ddb admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:09.922076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 927/1844] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:09 2022] PUT /identity/v3/projects/f0c6baba8aa248fdbaa02452ef21e987/users/a4dca1b96a7541698845aa6c79c71367/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:09.934336 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd78b2aa-0f15-4cad-bf17-4c376aa69058 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:09.934640 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd78b2aa-0f15-4cad-bf17-4c376aa69058 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:09.934908 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd78b2aa-0f15-4cad-bf17-4c376aa69058 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:09.978040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-fd78b2aa-0f15-4cad-bf17-4c376aa69058 None None] MFA Rules not processed for user `a4dca1b96a7541698845aa6c79c71367`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:09.979440 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fd78b2aa-0f15-4cad-bf17-4c376aa69058 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:10.145253 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:10.165629 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 918/1845] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:09 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:10.171033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458182f1-a263-4514-9376-312e1d979ddb admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:10.237761 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:10.263802 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458182f1-a263-4514-9376-312e1d979ddb admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:10.269835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bc7316-6318-4853-b549-050eeb694d17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:10.271079 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bc7316-6318-4853-b549-050eeb694d17 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:10.271496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bc7316-6318-4853-b549-050eeb694d17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:10.272276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bc7316-6318-4853-b549-050eeb694d17 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:10.273217 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bc7316-6318-4853-b549-050eeb694d17 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:10.275673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bc7316-6318-4853-b549-050eeb694d17 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:10.326198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 928/1846] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:10 2022] POST /identity/v3/projects => generated 383 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:10.327721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd78b2aa-0f15-4cad-bf17-4c376aa69058 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:10.350108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd78b2aa-0f15-4cad-bf17-4c376aa69058 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:10.446891 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fca964-3d70-413e-ae98-ccdeeeec1bae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:10.446891 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fca964-3d70-413e-ae98-ccdeeeec1bae admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:10.446891 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fca964-3d70-413e-ae98-ccdeeeec1bae admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:10.446891 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fca964-3d70-413e-ae98-ccdeeeec1bae admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:10.447344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8fca964-3d70-413e-ae98-ccdeeeec1bae admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:10.447344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8fca964-3d70-413e-ae98-ccdeeeec1bae admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:10.479304 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 919/1847] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:10 2022] POST /identity/v3/users => generated 298 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:10.483783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bc7316-6318-4853-b549-050eeb694d17 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:10.501369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bc7316-6318-4853-b549-050eeb694d17 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:10.512870 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746fd882-a920-4259-9bbe-428dc330fec4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:10.513756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746fd882-a920-4259-9bbe-428dc330fec4 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:10.514177 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746fd882-a920-4259-9bbe-428dc330fec4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:10.517170 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746fd882-a920-4259-9bbe-428dc330fec4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:10.517880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746fd882-a920-4259-9bbe-428dc330fec4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:10.519695 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746fd882-a920-4259-9bbe-428dc330fec4 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:10.520387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fca964-3d70-413e-ae98-ccdeeeec1bae admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:10.522511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b8fca964-3d70-413e-ae98-ccdeeeec1bae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:10.532376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 929/1848] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:10 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:10.535649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fca964-3d70-413e-ae98-ccdeeeec1bae admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:10.536586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746fd882-a920-4259-9bbe-428dc330fec4 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:10.542050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e1961d-6b1a-4850-9e8a-8b171749039a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:10.542050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e1961d-6b1a-4850-9e8a-8b171749039a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:10.542050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e1961d-6b1a-4850-9e8a-8b171749039a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:10.542331 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e1961d-6b1a-4850-9e8a-8b171749039a admin admin] PATH_INFO: `/v3/users/3344ced82dec4a5b950ac38a4c8c0d82` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:10.543393 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e1961d-6b1a-4850-9e8a-8b171749039a admin admin] RBAC: Authorizing `identity:delete_user(user_id=3344ced82dec4a5b950ac38a4c8c0d82)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:10.545465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e1961d-6b1a-4850-9e8a-8b171749039a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:10.549694 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746fd882-a920-4259-9bbe-428dc330fec4 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:10.560422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62878b2-72f3-43a6-aefc-8041b37e1759 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:10.561972 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62878b2-72f3-43a6-aefc-8041b37e1759 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:10.562264 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62878b2-72f3-43a6-aefc-8041b37e1759 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:10.562534 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62878b2-72f3-43a6-aefc-8041b37e1759 admin admin] PATH_INFO: `/v3/projects/6ecae603a45d42d4b787cbdccb4678d4/users/b1ac05cfe1e74a9da9201644dfc14f22/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:10.581509 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62878b2-72f3-43a6-aefc-8041b37e1759 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ecae603a45d42d4b787cbdccb4678d4, user_id=b1ac05cfe1e74a9da9201644dfc14f22, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:10.589475 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62878b2-72f3-43a6-aefc-8041b37e1759 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:10.604621 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 930/1849] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:10 2022] PUT /identity/v3/projects/6ecae603a45d42d4b787cbdccb4678d4/users/b1ac05cfe1e74a9da9201644dfc14f22/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:19:10.605674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62878b2-72f3-43a6-aefc-8041b37e1759 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:10.610631 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a0e1961d-6b1a-4850-9e8a-8b171749039a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3344ced82dec4a5b950ac38a4c8c0d82'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:10.611349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a0e1961d-6b1a-4850-9e8a-8b171749039a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3344ced82dec4a5b950ac38a4c8c0d82'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:10.625887 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a0e1961d-6b1a-4850-9e8a-8b171749039a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3344ced82dec4a5b950ac38a4c8c0d82'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:10.626555 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a0e1961d-6b1a-4850-9e8a-8b171749039a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3344ced82dec4a5b950ac38a4c8c0d82'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:10.634766 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a0e1961d-6b1a-4850-9e8a-8b171749039a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3344ced82dec4a5b950ac38a4c8c0d82'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:10.662415 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 920/1850] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:10 2022] DELETE /identity/v3/users/3344ced82dec4a5b950ac38a4c8c0d82 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:19:10.685036 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aa19c6-6f67-4b42-91c8-81bf8e90d3cf None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:10.685479 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aa19c6-6f67-4b42-91c8-81bf8e90d3cf None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:10.685906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aa19c6-6f67-4b42-91c8-81bf8e90d3cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:10.719848 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62878b2-72f3-43a6-aefc-8041b37e1759 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:10.727727 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-74aa19c6-6f67-4b42-91c8-81bf8e90d3cf None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:10.729298 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-74aa19c6-6f67-4b42-91c8-81bf8e90d3cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:10.807979 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792fa5e0-ba8a-44be-adf2-c52e1e3193c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:10.808973 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792fa5e0-ba8a-44be-adf2-c52e1e3193c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:10.809261 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792fa5e0-ba8a-44be-adf2-c52e1e3193c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:10.809633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792fa5e0-ba8a-44be-adf2-c52e1e3193c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:10.810895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792fa5e0-ba8a-44be-adf2-c52e1e3193c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:10.818805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792fa5e0-ba8a-44be-adf2-c52e1e3193c7 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:10.831054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 931/1851] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:10 2022] GET /identity/v3/roles => generated 1420 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:10.837203 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792fa5e0-ba8a-44be-adf2-c52e1e3193c7 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:10.842898 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-792fa5e0-ba8a-44be-adf2-c52e1e3193c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:10.861617 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792fa5e0-ba8a-44be-adf2-c52e1e3193c7 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:10.866632 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60efcebd-cabb-428c-a4df-ff5c305640f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:10.867428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60efcebd-cabb-428c-a4df-ff5c305640f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:10.867772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60efcebd-cabb-428c-a4df-ff5c305640f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:10.868110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60efcebd-cabb-428c-a4df-ff5c305640f7 admin admin] PATH_INFO: `/v3/projects/6ecae603a45d42d4b787cbdccb4678d4/users/b1ac05cfe1e74a9da9201644dfc14f22/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:10.870290 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60efcebd-cabb-428c-a4df-ff5c305640f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ecae603a45d42d4b787cbdccb4678d4, user_id=b1ac05cfe1e74a9da9201644dfc14f22, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:10.873215 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60efcebd-cabb-428c-a4df-ff5c305640f7 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:10.891984 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 921/1852] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:10.893133 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74aa19c6-6f67-4b42-91c8-81bf8e90d3cf None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:10.894630 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-74aa19c6-6f67-4b42-91c8-81bf8e90d3cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:10.899016 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 932/1853] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:10 2022] PUT /identity/v3/projects/6ecae603a45d42d4b787cbdccb4678d4/users/b1ac05cfe1e74a9da9201644dfc14f22/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:10.902035 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:10.911204 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c09f4b6-d663-4844-a9f1-cd0816e84667 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:10.914487 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c09f4b6-d663-4844-a9f1-cd0816e84667 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:10.914487 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c09f4b6-d663-4844-a9f1-cd0816e84667 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:10.973963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-0c09f4b6-d663-4844-a9f1-cd0816e84667 None None] MFA Rules not processed for user `b1ac05cfe1e74a9da9201644dfc14f22`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:10.975246 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0c09f4b6-d663-4844-a9f1-cd0816e84667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:10.995391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:10.999379 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74aa19c6-6f67-4b42-91c8-81bf8e90d3cf None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:11.084245 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac378678-be22-4b7b-ac8c-da2e55c1949d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:11.084876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac378678-be22-4b7b-ac8c-da2e55c1949d service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:11.085156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac378678-be22-4b7b-ac8c-da2e55c1949d service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:11.085499 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac378678-be22-4b7b-ac8c-da2e55c1949d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:11.119728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 933/1854] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:10 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:11.128677 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c09f4b6-d663-4844-a9f1-cd0816e84667 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:11.134876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0c09f4b6-d663-4844-a9f1-cd0816e84667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:11.208524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac378678-be22-4b7b-ac8c-da2e55c1949d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:11.211985 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac378678-be22-4b7b-ac8c-da2e55c1949d service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:11.219706 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c09f4b6-d663-4844-a9f1-cd0816e84667 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:11.225685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc517ce9-10e0-42eb-aec4-927530fbb4b5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:11.226578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc517ce9-10e0-42eb-aec4-927530fbb4b5 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:11.226975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc517ce9-10e0-42eb-aec4-927530fbb4b5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:11.227760 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 922/1855] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:11.229011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc517ce9-10e0-42eb-aec4-927530fbb4b5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:11.233890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-cc517ce9-10e0-42eb-aec4-927530fbb4b5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:11.330095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc517ce9-10e0-42eb-aec4-927530fbb4b5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:11.333339 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc517ce9-10e0-42eb-aec4-927530fbb4b5 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:11.345284 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 934/1856] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:11 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:11.836699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac378678-be22-4b7b-ac8c-da2e55c1949d service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:11.838351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ac378678-be22-4b7b-ac8c-da2e55c1949d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:11.900866 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac378678-be22-4b7b-ac8c-da2e55c1949d service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:11.905776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c09c4f0-7d0d-4e2d-8567-49bcdf4cad92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:11.906197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c09c4f0-7d0d-4e2d-8567-49bcdf4cad92 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:11.906472 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c09c4f0-7d0d-4e2d-8567-49bcdf4cad92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:11.906741 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c09c4f0-7d0d-4e2d-8567-49bcdf4cad92 admin admin] PATH_INFO: `/v3/projects/68ec334ca1994b729290d6acef7e58ae` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:11.908480 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c09c4f0-7d0d-4e2d-8567-49bcdf4cad92 admin admin] RBAC: Authorizing `identity:delete_project(project_id=68ec334ca1994b729290d6acef7e58ae)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:11.909882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c09c4f0-7d0d-4e2d-8567-49bcdf4cad92 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:12.010105 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1c09c4f0-7d0d-4e2d-8567-49bcdf4cad92 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '68ec334ca1994b729290d6acef7e58ae', 'request_id': 'req-1c09c4f0-7d0d-4e2d-8567-49bcdf4cad92'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:12.016037 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1c09c4f0-7d0d-4e2d-8567-49bcdf4cad92 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '68ec334ca1994b729290d6acef7e58ae', 'request_id': 'req-1c09c4f0-7d0d-4e2d-8567-49bcdf4cad92'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:12.022381 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1c09c4f0-7d0d-4e2d-8567-49bcdf4cad92 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '68ec334ca1994b729290d6acef7e58ae', 'request_id': 'req-1c09c4f0-7d0d-4e2d-8567-49bcdf4cad92'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:12.028437 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:12.040341 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1c09c4f0-7d0d-4e2d-8567-49bcdf4cad92 admin admin] The token cache is being invalidate because project 68ec334ca1994b729290d6acef7e58ae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:12.040726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1c09c4f0-7d0d-4e2d-8567-49bcdf4cad92 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:12.042011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1c09c4f0-7d0d-4e2d-8567-49bcdf4cad92 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:12.044021 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 923/1857] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:11 2022] DELETE /identity/v3/projects/68ec334ca1994b729290d6acef7e58ae => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:12.051277 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc517ce9-10e0-42eb-aec4-927530fbb4b5 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:12.065300 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc517ce9-10e0-42eb-aec4-927530fbb4b5 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:12.137816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d909f8-b924-4934-bdf4-a9e0febf0726 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:12.138342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d909f8-b924-4934-bdf4-a9e0febf0726 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:12.138737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d909f8-b924-4934-bdf4-a9e0febf0726 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:12.139050 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d909f8-b924-4934-bdf4-a9e0febf0726 admin admin] PATH_INFO: `/v3/users/46bc96728c3144b7be8cf1947381b359` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:12.140328 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d909f8-b924-4934-bdf4-a9e0febf0726 admin admin] RBAC: Authorizing `identity:delete_user(user_id=46bc96728c3144b7be8cf1947381b359)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:12.142260 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d909f8-b924-4934-bdf4-a9e0febf0726 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:12.157475 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:12.198560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d1d909f8-b924-4934-bdf4-a9e0febf0726 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '46bc96728c3144b7be8cf1947381b359'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:12.199931 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d1d909f8-b924-4934-bdf4-a9e0febf0726 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '46bc96728c3144b7be8cf1947381b359'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:12.208277 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d1d909f8-b924-4934-bdf4-a9e0febf0726 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '46bc96728c3144b7be8cf1947381b359'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:12.208968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d1d909f8-b924-4934-bdf4-a9e0febf0726 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '46bc96728c3144b7be8cf1947381b359'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:12.222042 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d1d909f8-b924-4934-bdf4-a9e0febf0726 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '46bc96728c3144b7be8cf1947381b359'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:12.235070 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 935/1858] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:12 2022] DELETE /identity/v3/users/46bc96728c3144b7be8cf1947381b359 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:19:12.607205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c09c4f0-7d0d-4e2d-8567-49bcdf4cad92 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:12.607205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1c09c4f0-7d0d-4e2d-8567-49bcdf4cad92 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:12.618565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c09c4f0-7d0d-4e2d-8567-49bcdf4cad92 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:12.676790 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9df1565-3928-48d7-8aeb-b6cf9d65a125 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:12.677340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9df1565-3928-48d7-8aeb-b6cf9d65a125 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:12.677616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9df1565-3928-48d7-8aeb-b6cf9d65a125 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:12.677893 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9df1565-3928-48d7-8aeb-b6cf9d65a125 admin admin] PATH_INFO: `/v3/projects/027af2c11f494e59a4cbbd6158ed3b64` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:12.679035 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9df1565-3928-48d7-8aeb-b6cf9d65a125 admin admin] RBAC: Authorizing `identity:delete_project(project_id=027af2c11f494e59a4cbbd6158ed3b64)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:12.680936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9df1565-3928-48d7-8aeb-b6cf9d65a125 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:12.728708 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f9df1565-3928-48d7-8aeb-b6cf9d65a125 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '027af2c11f494e59a4cbbd6158ed3b64', 'request_id': 'req-f9df1565-3928-48d7-8aeb-b6cf9d65a125'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:12.734826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f9df1565-3928-48d7-8aeb-b6cf9d65a125 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '027af2c11f494e59a4cbbd6158ed3b64', 'request_id': 'req-f9df1565-3928-48d7-8aeb-b6cf9d65a125'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:12.741297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f9df1565-3928-48d7-8aeb-b6cf9d65a125 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '027af2c11f494e59a4cbbd6158ed3b64', 'request_id': 'req-f9df1565-3928-48d7-8aeb-b6cf9d65a125'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:12.753821 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f9df1565-3928-48d7-8aeb-b6cf9d65a125 admin admin] The token cache is being invalidate because project 027af2c11f494e59a4cbbd6158ed3b64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:12.754108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f9df1565-3928-48d7-8aeb-b6cf9d65a125 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:12.754722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f9df1565-3928-48d7-8aeb-b6cf9d65a125 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:12.756228 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 924/1859] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:12 2022] DELETE /identity/v3/projects/027af2c11f494e59a4cbbd6158ed3b64 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:12.761111 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2eaaa5-8962-48eb-8677-0b30f84fa30e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:12.761400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2eaaa5-8962-48eb-8677-0b30f84fa30e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:12.761676 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2eaaa5-8962-48eb-8677-0b30f84fa30e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:12.781596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7f2eaaa5-8962-48eb-8677-0b30f84fa30e None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:12.782803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7f2eaaa5-8962-48eb-8677-0b30f84fa30e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:12.841679 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 936/1860] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:12.850092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d188196-10c0-4c97-8f92-fd0d6da0d4ce None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:12.850382 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d188196-10c0-4c97-8f92-fd0d6da0d4ce None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:12.850660 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d188196-10c0-4c97-8f92-fd0d6da0d4ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:12.871528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8d188196-10c0-4c97-8f92-fd0d6da0d4ce None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:12.872720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8d188196-10c0-4c97-8f92-fd0d6da0d4ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:12.880504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 925/1861] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:12.884755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2eaaa5-8962-48eb-8677-0b30f84fa30e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:12.894799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2eaaa5-8962-48eb-8677-0b30f84fa30e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:12.899348 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9437639-a90a-4de7-aa91-e2f899ccd42b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:12.899942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9437639-a90a-4de7-aa91-e2f899ccd42b admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:12.900217 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9437639-a90a-4de7-aa91-e2f899ccd42b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:12.900487 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9437639-a90a-4de7-aa91-e2f899ccd42b admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:12.901125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9437639-a90a-4de7-aa91-e2f899ccd42b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:19:12.901425 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9437639-a90a-4de7-aa91-e2f899ccd42b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:12.903043 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9437639-a90a-4de7-aa91-e2f899ccd42b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:12.914475 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 937/1862] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:19:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:19:12.918626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d188196-10c0-4c97-8f92-fd0d6da0d4ce None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:12.928326 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d188196-10c0-4c97-8f92-fd0d6da0d4ce None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:12.932754 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12298e6-667a-4ebe-b8b2-376280af3d18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:12.933375 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12298e6-667a-4ebe-b8b2-376280af3d18 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:12.933650 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12298e6-667a-4ebe-b8b2-376280af3d18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:12.933919 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12298e6-667a-4ebe-b8b2-376280af3d18 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:12.934576 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12298e6-667a-4ebe-b8b2-376280af3d18 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:12.937392 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12298e6-667a-4ebe-b8b2-376280af3d18 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:12.963725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 926/1863] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:12 2022] POST /identity/v3/projects => generated 377 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:12.968293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9437639-a90a-4de7-aa91-e2f899ccd42b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:12.978097 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9437639-a90a-4de7-aa91-e2f899ccd42b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:13.033321 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9423060-e158-4e52-81cc-9ce5d3171e63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:13.033927 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9423060-e158-4e52-81cc-9ce5d3171e63 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:13.034204 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9423060-e158-4e52-81cc-9ce5d3171e63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:13.034476 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9423060-e158-4e52-81cc-9ce5d3171e63 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:13.035153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9423060-e158-4e52-81cc-9ce5d3171e63 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:13.037036 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9423060-e158-4e52-81cc-9ce5d3171e63 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:13.059703 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:13.060326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 938/1864] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:12 2022] POST /identity/v3/users => generated 289 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:13.064013 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12298e6-667a-4ebe-b8b2-376280af3d18 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:13.074596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12298e6-667a-4ebe-b8b2-376280af3d18 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:13.079180 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d2909a-e802-45c7-9314-b7e709fe81fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:13.079787 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d2909a-e802-45c7-9314-b7e709fe81fb admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:13.080063 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d2909a-e802-45c7-9314-b7e709fe81fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:13.080339 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d2909a-e802-45c7-9314-b7e709fe81fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:13.080973 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d2909a-e802-45c7-9314-b7e709fe81fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:13.082554 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d2909a-e802-45c7-9314-b7e709fe81fb admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:13.093332 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 927/1865] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:13 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:19:13.097339 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9423060-e158-4e52-81cc-9ce5d3171e63 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:13.111783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9423060-e158-4e52-81cc-9ce5d3171e63 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:13.117606 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675c0641-4b57-4e13-9019-63cd8560e016 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:13.118322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675c0641-4b57-4e13-9019-63cd8560e016 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:13.118670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675c0641-4b57-4e13-9019-63cd8560e016 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:13.118968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675c0641-4b57-4e13-9019-63cd8560e016 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:13.119701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675c0641-4b57-4e13-9019-63cd8560e016 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:13.121287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675c0641-4b57-4e13-9019-63cd8560e016 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:13.131845 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 939/1866] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:13 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:13.136070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d2909a-e802-45c7-9314-b7e709fe81fb admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:13.145850 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d2909a-e802-45c7-9314-b7e709fe81fb admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:13.150825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c933f78-de70-4f52-9a20-847082bb684c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:13.151603 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c933f78-de70-4f52-9a20-847082bb684c admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:13.151895 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c933f78-de70-4f52-9a20-847082bb684c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:13.152167 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c933f78-de70-4f52-9a20-847082bb684c admin admin] PATH_INFO: `/v3/projects/e026e246277240e5b2606cd383148a5b/users/1c5e4469a44446bbb87743271fb4b12a/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:13.168510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c933f78-de70-4f52-9a20-847082bb684c admin admin] RBAC: Authorizing `identity:create_grant(project_id=e026e246277240e5b2606cd383148a5b, user_id=1c5e4469a44446bbb87743271fb4b12a, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:13.171881 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c933f78-de70-4f52-9a20-847082bb684c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:13.185596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 928/1867] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:13 2022] PUT /identity/v3/projects/e026e246277240e5b2606cd383148a5b/users/1c5e4469a44446bbb87743271fb4b12a/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:13.196309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0110928-9d9f-4b6b-b788-531414efcc81 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:13.196598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0110928-9d9f-4b6b-b788-531414efcc81 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:13.196873 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0110928-9d9f-4b6b-b788-531414efcc81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:13.225133 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e0110928-9d9f-4b6b-b788-531414efcc81 None None] MFA Rules not processed for user `1c5e4469a44446bbb87743271fb4b12a`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:13.226338 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e0110928-9d9f-4b6b-b788-531414efcc81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:13.292440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 940/1868] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:13 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:13.296985 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c933f78-de70-4f52-9a20-847082bb684c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:13.317247 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:INSERT=4,keystone:DELETE=8,keystone:SELECT=88 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:13.363681 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c933f78-de70-4f52-9a20-847082bb684c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:13.368329 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca290f1-0e22-411b-8f72-c7613c0e167d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:13.368935 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca290f1-0e22-411b-8f72-c7613c0e167d admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:13.369214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca290f1-0e22-411b-8f72-c7613c0e167d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:13.369484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca290f1-0e22-411b-8f72-c7613c0e167d admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:13.370102 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca290f1-0e22-411b-8f72-c7613c0e167d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:13.372027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca290f1-0e22-411b-8f72-c7613c0e167d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:13.398087 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 929/1869] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:13 2022] POST /identity/v3/projects => generated 377 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:13.402125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0110928-9d9f-4b6b-b788-531414efcc81 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:13.413838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0110928-9d9f-4b6b-b788-531414efcc81 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:13.472509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685633aa-246a-48f4-ae62-bb92ed1cc194 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:13.472953 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685633aa-246a-48f4-ae62-bb92ed1cc194 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:13.473300 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685633aa-246a-48f4-ae62-bb92ed1cc194 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:13.481242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712a275b-72dc-4ff0-a1ee-c09c014c8301 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:13.481996 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712a275b-72dc-4ff0-a1ee-c09c014c8301 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:13.482274 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712a275b-72dc-4ff0-a1ee-c09c014c8301 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:13.482600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712a275b-72dc-4ff0-a1ee-c09c014c8301 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:13.483282 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712a275b-72dc-4ff0-a1ee-c09c014c8301 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:13.485303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712a275b-72dc-4ff0-a1ee-c09c014c8301 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:13.510998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-685633aa-246a-48f4-ae62-bb92ed1cc194 None None] MFA Rules not processed for user `09cd263f72484733adaaa0850ebfa609`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:13.512524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-685633aa-246a-48f4-ae62-bb92ed1cc194 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:13.533769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 941/1870] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:13 2022] POST /identity/v3/users => generated 289 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:13.537865 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712a275b-72dc-4ff0-a1ee-c09c014c8301 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:13.549670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712a275b-72dc-4ff0-a1ee-c09c014c8301 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:13.554834 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b377c0fb-3d24-47ac-835d-65bb200f0051 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:13.555229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b377c0fb-3d24-47ac-835d-65bb200f0051 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:13.555622 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b377c0fb-3d24-47ac-835d-65bb200f0051 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:13.555895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b377c0fb-3d24-47ac-835d-65bb200f0051 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:13.556540 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b377c0fb-3d24-47ac-835d-65bb200f0051 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:13.558148 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b377c0fb-3d24-47ac-835d-65bb200f0051 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:13.569478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 942/1871] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:13 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:13.573282 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b377c0fb-3d24-47ac-835d-65bb200f0051 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:13.579851 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 930/1872] 10.222.0.25 () {58 vars in 1015 bytes} [Tue Sep 13 01:19:13 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:13.584028 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b377c0fb-3d24-47ac-835d-65bb200f0051 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:13.588708 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d313565-be5a-4a0f-ab77-c6854953d3a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:13.589235 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d313565-be5a-4a0f-ab77-c6854953d3a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:13.589516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d313565-be5a-4a0f-ab77-c6854953d3a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:13.589798 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d313565-be5a-4a0f-ab77-c6854953d3a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:13.590444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d313565-be5a-4a0f-ab77-c6854953d3a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:13.592191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d313565-be5a-4a0f-ab77-c6854953d3a0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:13.603961 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 943/1873] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:13 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:19:13.608185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685633aa-246a-48f4-ae62-bb92ed1cc194 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:13.620493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685633aa-246a-48f4-ae62-bb92ed1cc194 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:13.626311 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013ac708-7b84-402e-8108-820f1f1e4432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:13.627020 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013ac708-7b84-402e-8108-820f1f1e4432 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:13.627307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013ac708-7b84-402e-8108-820f1f1e4432 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:13.627584 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013ac708-7b84-402e-8108-820f1f1e4432 admin admin] PATH_INFO: `/v3/projects/041f608de8e34c8bac1cede0693f7112/users/2cc353987db54592ae437db64595db52/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:13.645372 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013ac708-7b84-402e-8108-820f1f1e4432 admin admin] RBAC: Authorizing `identity:create_grant(project_id=041f608de8e34c8bac1cede0693f7112, user_id=2cc353987db54592ae437db64595db52, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:13.649447 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013ac708-7b84-402e-8108-820f1f1e4432 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:13.667517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 931/1874] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:13 2022] PUT /identity/v3/projects/041f608de8e34c8bac1cede0693f7112/users/2cc353987db54592ae437db64595db52/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:13.678981 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8451719-b483-408a-bc37-b2f0969aa6c1 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:13.679340 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8451719-b483-408a-bc37-b2f0969aa6c1 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:13.679727 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8451719-b483-408a-bc37-b2f0969aa6c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:13.711408 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c8451719-b483-408a-bc37-b2f0969aa6c1 None None] MFA Rules not processed for user `2cc353987db54592ae437db64595db52`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:13.712582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c8451719-b483-408a-bc37-b2f0969aa6c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:13.742558 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013ac708-7b84-402e-8108-820f1f1e4432 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:13.744689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-013ac708-7b84-402e-8108-820f1f1e4432 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:13.776694 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:13.845155 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013ac708-7b84-402e-8108-820f1f1e4432 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:13.858707 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2381ddc4-9b10-447a-a8a2-816ec39d6e8a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:13.859165 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2381ddc4-9b10-447a-a8a2-816ec39d6e8a service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:13.859507 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2381ddc4-9b10-447a-a8a2-816ec39d6e8a service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:13.859713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2381ddc4-9b10-447a-a8a2-816ec39d6e8a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:13.866818 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 944/1875] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:13 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:13.866818 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8451719-b483-408a-bc37-b2f0969aa6c1 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:13.867964 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2381ddc4-9b10-447a-a8a2-816ec39d6e8a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:13.869329 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2381ddc4-9b10-447a-a8a2-816ec39d6e8a service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:13.881301 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 932/1876] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:13 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:13.882168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2381ddc4-9b10-447a-a8a2-816ec39d6e8a service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:13.883674 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2381ddc4-9b10-447a-a8a2-816ec39d6e8a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:13.897282 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2381ddc4-9b10-447a-a8a2-816ec39d6e8a service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:13.908864 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724e3983-56bd-409f-8bfe-a09c54e9612c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:13.910173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724e3983-56bd-409f-8bfe-a09c54e9612c service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:13.911048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724e3983-56bd-409f-8bfe-a09c54e9612c service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:13.911382 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724e3983-56bd-409f-8bfe-a09c54e9612c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:13.912371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8451719-b483-408a-bc37-b2f0969aa6c1 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:13.920282 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c11939-8276-4700-b26a-1e90cc02a3a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:13.921520 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c11939-8276-4700-b26a-1e90cc02a3a9 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:13.921956 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c11939-8276-4700-b26a-1e90cc02a3a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:13.922426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c11939-8276-4700-b26a-1e90cc02a3a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:13.931424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c11939-8276-4700-b26a-1e90cc02a3a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:13.932870 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c11939-8276-4700-b26a-1e90cc02a3a9 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:13.942541 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 945/1877] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:13 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:13.943442 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c11939-8276-4700-b26a-1e90cc02a3a9 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:13.951320 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724e3983-56bd-409f-8bfe-a09c54e9612c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:13.956152 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724e3983-56bd-409f-8bfe-a09c54e9612c service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:13.971364 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 933/1878] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:13 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:14.067415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c11939-8276-4700-b26a-1e90cc02a3a9 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:14.076468 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462c2ecb-0073-4af3-804f-0e16e679c321 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:14.077921 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462c2ecb-0073-4af3-804f-0e16e679c321 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:14.078309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462c2ecb-0073-4af3-804f-0e16e679c321 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:14.078691 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462c2ecb-0073-4af3-804f-0e16e679c321 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:14.080724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462c2ecb-0073-4af3-804f-0e16e679c321 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:14.083712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462c2ecb-0073-4af3-804f-0e16e679c321 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:14.123580 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 946/1879] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:13 2022] POST /identity/v3/projects => generated 377 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:14.128228 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724e3983-56bd-409f-8bfe-a09c54e9612c service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:14.162465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724e3983-56bd-409f-8bfe-a09c54e9612c service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:14.197903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:14.286018 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5b2ad6-2e45-47b2-aef1-1a75b2405597 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:14.286706 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5b2ad6-2e45-47b2-aef1-1a75b2405597 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:14.289794 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5b2ad6-2e45-47b2-aef1-1a75b2405597 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:14.289794 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5b2ad6-2e45-47b2-aef1-1a75b2405597 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:14.289794 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5b2ad6-2e45-47b2-aef1-1a75b2405597 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:14.291637 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5b2ad6-2e45-47b2-aef1-1a75b2405597 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:14.363461 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 934/1880] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:14 2022] POST /identity/v3/users => generated 295 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:14.367459 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462c2ecb-0073-4af3-804f-0e16e679c321 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:14.392802 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462c2ecb-0073-4af3-804f-0e16e679c321 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:14.402721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59880900-13f7-4a1c-a931-3ebf826be066 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:14.403587 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59880900-13f7-4a1c-a931-3ebf826be066 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:14.404013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59880900-13f7-4a1c-a931-3ebf826be066 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:14.404408 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59880900-13f7-4a1c-a931-3ebf826be066 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:14.405305 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59880900-13f7-4a1c-a931-3ebf826be066 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:14.407065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59880900-13f7-4a1c-a931-3ebf826be066 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:14.442918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 947/1881] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:14 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:14.447307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5b2ad6-2e45-47b2-aef1-1a75b2405597 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:14.473975 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5b2ad6-2e45-47b2-aef1-1a75b2405597 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:14.503447 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7080b8a-35f4-4341-9fdd-4f3c23df2087 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:14.505494 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7080b8a-35f4-4341-9fdd-4f3c23df2087 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:14.505494 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7080b8a-35f4-4341-9fdd-4f3c23df2087 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:14.505731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7080b8a-35f4-4341-9fdd-4f3c23df2087 admin admin] PATH_INFO: `/v3/projects/cb38177fb9b445859cd177334a1d2c7a/users/acb01352910e4d228874d13e06d66ef1/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:14.569302 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7080b8a-35f4-4341-9fdd-4f3c23df2087 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb38177fb9b445859cd177334a1d2c7a, user_id=acb01352910e4d228874d13e06d66ef1, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:14.583713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7080b8a-35f4-4341-9fdd-4f3c23df2087 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:14.634494 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 935/1882] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:14 2022] PUT /identity/v3/projects/cb38177fb9b445859cd177334a1d2c7a/users/acb01352910e4d228874d13e06d66ef1/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:14.638413 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59880900-13f7-4a1c-a931-3ebf826be066 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:14.656604 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59880900-13f7-4a1c-a931-3ebf826be066 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:14.789150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acfe6e0-77d9-4ec3-949f-61b20a9222f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:14.789914 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acfe6e0-77d9-4ec3-949f-61b20a9222f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:14.790281 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acfe6e0-77d9-4ec3-949f-61b20a9222f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:14.790742 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acfe6e0-77d9-4ec3-949f-61b20a9222f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:14.791546 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0acfe6e0-77d9-4ec3-949f-61b20a9222f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:14.793371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0acfe6e0-77d9-4ec3-949f-61b20a9222f5 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:14.807477 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 948/1883] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:14 2022] GET /identity/v3/roles => generated 1420 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:14.818533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7080b8a-35f4-4341-9fdd-4f3c23df2087 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:14.833885 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7080b8a-35f4-4341-9fdd-4f3c23df2087 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:14.840902 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f320b333-384b-472d-a1c2-cc02a36c6a3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:14.842046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f320b333-384b-472d-a1c2-cc02a36c6a3b admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:14.842701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f320b333-384b-472d-a1c2-cc02a36c6a3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:14.843209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f320b333-384b-472d-a1c2-cc02a36c6a3b admin admin] PATH_INFO: `/v3/projects/cb38177fb9b445859cd177334a1d2c7a/users/acb01352910e4d228874d13e06d66ef1/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:14.851555 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f320b333-384b-472d-a1c2-cc02a36c6a3b admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb38177fb9b445859cd177334a1d2c7a, user_id=acb01352910e4d228874d13e06d66ef1, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:14.856830 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f320b333-384b-472d-a1c2-cc02a36c6a3b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:14.879925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 936/1884] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:14 2022] PUT /identity/v3/projects/cb38177fb9b445859cd177334a1d2c7a/users/acb01352910e4d228874d13e06d66ef1/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:19:14.890462 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2855e1-681a-4973-9b67-f1e793496990 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:14.890758 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2855e1-681a-4973-9b67-f1e793496990 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:14.891061 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2855e1-681a-4973-9b67-f1e793496990 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:14.970189 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8d2855e1-681a-4973-9b67-f1e793496990 None None] MFA Rules not processed for user `acb01352910e4d228874d13e06d66ef1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:14.973514 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8d2855e1-681a-4973-9b67-f1e793496990 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:15.007589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:15.083218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 949/1885] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:14 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:15.096990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f320b333-384b-472d-a1c2-cc02a36c6a3b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:15.098881 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f320b333-384b-472d-a1c2-cc02a36c6a3b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:15.213706 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f320b333-384b-472d-a1c2-cc02a36c6a3b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:15.218686 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6405d5e7-434d-440a-9693-6879380695e0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:15.219426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6405d5e7-434d-440a-9693-6879380695e0 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:15.219426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6405d5e7-434d-440a-9693-6879380695e0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:15.220053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6405d5e7-434d-440a-9693-6879380695e0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:15.359376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6405d5e7-434d-440a-9693-6879380695e0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:15.360992 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6405d5e7-434d-440a-9693-6879380695e0 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:15.372785 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 937/1886] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:15 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 276 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:15.616789 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6834aa-f257-4683-b8bc-e4d1c546cc10 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:15.617790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6834aa-f257-4683-b8bc-e4d1c546cc10 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:15.618360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6834aa-f257-4683-b8bc-e4d1c546cc10 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:15.620687 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 950/1887] 10.222.0.15 () {54 vars in 921 bytes} [Tue Sep 13 01:19:15 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 01:19:15.682275 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52edc22b-06a0-4f55-87d0-73c50ae26673 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:15.682948 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52edc22b-06a0-4f55-87d0-73c50ae26673 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:15.682948 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52edc22b-06a0-4f55-87d0-73c50ae26673 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:15.716271 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-52edc22b-06a0-4f55-87d0-73c50ae26673 None None] MFA Rules not processed for user `2f2da4d2d3604d16b58dc25b8c17c49a`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:15.717628 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-52edc22b-06a0-4f55-87d0-73c50ae26673 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:15.742652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:15.825102 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 938/1888] 10.222.0.15 () {58 vars in 1018 bytes} [Tue Sep 13 01:19:15 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:15.927302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6834aa-f257-4683-b8bc-e4d1c546cc10 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:15.931392 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-df6834aa-f257-4683-b8bc-e4d1c546cc10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:15.984954 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6834aa-f257-4683-b8bc-e4d1c546cc10 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:16.006085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a99cdf-0b15-475c-9516-2913693b073e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:16.006830 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a99cdf-0b15-475c-9516-2913693b073e service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:16.007248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a99cdf-0b15-475c-9516-2913693b073e service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:16.007616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a99cdf-0b15-475c-9516-2913693b073e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:16.028975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a99cdf-0b15-475c-9516-2913693b073e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:16.033798 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a99cdf-0b15-475c-9516-2913693b073e service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:16.062522 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 951/1889] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:17.398734 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52edc22b-06a0-4f55-87d0-73c50ae26673 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:17.400207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-52edc22b-06a0-4f55-87d0-73c50ae26673 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:17.418213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52edc22b-06a0-4f55-87d0-73c50ae26673 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:17.422400 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1a5fab-ba31-4d39-b419-d71452e8237c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:17.423403 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1a5fab-ba31-4d39-b419-d71452e8237c service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:17.424572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1a5fab-ba31-4d39-b419-d71452e8237c service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:17.424996 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1a5fab-ba31-4d39-b419-d71452e8237c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:17.433613 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71e5f88-0a08-450b-92ad-256391c08562 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:17.434110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71e5f88-0a08-450b-92ad-256391c08562 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:17.434201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71e5f88-0a08-450b-92ad-256391c08562 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:17.436625 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 952/1890] 10.222.0.15 () {54 vars in 921 bytes} [Tue Sep 13 01:19:17 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:19:17.460274 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cb39cc-e709-4ffe-bf45-23ce1da1cb34 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:17.461348 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cb39cc-e709-4ffe-bf45-23ce1da1cb34 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:17.461807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cb39cc-e709-4ffe-bf45-23ce1da1cb34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:17.485270 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-97cb39cc-e709-4ffe-bf45-23ce1da1cb34 None None] MFA Rules not processed for user `2f2da4d2d3604d16b58dc25b8c17c49a`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:17.486621 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-97cb39cc-e709-4ffe-bf45-23ce1da1cb34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:17.497255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 953/1891] 10.222.0.15 () {58 vars in 1018 bytes} [Tue Sep 13 01:19:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:17.498987 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cb39cc-e709-4ffe-bf45-23ce1da1cb34 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:17.500522 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1a5fab-ba31-4d39-b419-d71452e8237c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:17.501971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1a5fab-ba31-4d39-b419-d71452e8237c service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:17.510024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cb39cc-e709-4ffe-bf45-23ce1da1cb34 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:17.512891 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 939/1892] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:17 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:17.515465 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f657a1-ac31-4fec-9525-3aaa1d355b72 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:17.516085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f657a1-ac31-4fec-9525-3aaa1d355b72 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:17.516451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f657a1-ac31-4fec-9525-3aaa1d355b72 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:17.516727 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f657a1-ac31-4fec-9525-3aaa1d355b72 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:17.518538 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c8f657a1-ac31-4fec-9525-3aaa1d355b72 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:17.520535 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1a5fab-ba31-4d39-b419-d71452e8237c service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:17.521964 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5d1a5fab-ba31-4d39-b419-d71452e8237c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:17.568183 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1a5fab-ba31-4d39-b419-d71452e8237c service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:17.586204 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e024ac-a1d1-4034-8899-7479425cabe5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:17.587203 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e024ac-a1d1-4034-8899-7479425cabe5 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:17.587645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e024ac-a1d1-4034-8899-7479425cabe5 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:17.588079 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e024ac-a1d1-4034-8899-7479425cabe5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:17.606094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e024ac-a1d1-4034-8899-7479425cabe5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:17.607786 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e024ac-a1d1-4034-8899-7479425cabe5 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:17.631662 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 940/1893] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:19:17.646787 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f657a1-ac31-4fec-9525-3aaa1d355b72 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:17.649820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f657a1-ac31-4fec-9525-3aaa1d355b72 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:17.665893 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 954/1894] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:17 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:19.858770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e024ac-a1d1-4034-8899-7479425cabe5 service nova] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:19.868364 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e024ac-a1d1-4034-8899-7479425cabe5 service nova] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:19.872352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5f5e40-3dca-4320-928d-498ba132c588 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:19.872946 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5f5e40-3dca-4320-928d-498ba132c588 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:19.873221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5f5e40-3dca-4320-928d-498ba132c588 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:19.874009 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5f5e40-3dca-4320-928d-498ba132c588 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:19.883491 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5f5e40-3dca-4320-928d-498ba132c588 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:19.884959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5f5e40-3dca-4320-928d-498ba132c588 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:19.902119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 941/1895] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:19 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:24.053255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f657a1-ac31-4fec-9525-3aaa1d355b72 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:24.096836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5f5e40-3dca-4320-928d-498ba132c588 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:24.098267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ef5f5e40-3dca-4320-928d-498ba132c588 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:24.180181 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f657a1-ac31-4fec-9525-3aaa1d355b72 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:24.195612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4811af18-6f79-4b09-b876-05bc943ad1fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:24.196690 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4811af18-6f79-4b09-b876-05bc943ad1fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:24.196690 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4811af18-6f79-4b09-b876-05bc943ad1fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:24.196883 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4811af18-6f79-4b09-b876-05bc943ad1fd admin admin] PATH_INFO: `/v3/users/1c5e4469a44446bbb87743271fb4b12a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:24.202503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4811af18-6f79-4b09-b876-05bc943ad1fd admin admin] RBAC: Authorizing `identity:delete_user(user_id=1c5e4469a44446bbb87743271fb4b12a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:24.209287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4811af18-6f79-4b09-b876-05bc943ad1fd admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:24.236825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5f5e40-3dca-4320-928d-498ba132c588 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:24.244464 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5fa3dd-269c-496f-82e2-659889fe29ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:24.245404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5fa3dd-269c-496f-82e2-659889fe29ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:24.246671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5fa3dd-269c-496f-82e2-659889fe29ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:24.247307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5fa3dd-269c-496f-82e2-659889fe29ad admin admin] PATH_INFO: `/v3/users/a4dca1b96a7541698845aa6c79c71367` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:24.249984 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5fa3dd-269c-496f-82e2-659889fe29ad admin admin] RBAC: Authorizing `identity:delete_user(user_id=a4dca1b96a7541698845aa6c79c71367)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:24.252447 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5fa3dd-269c-496f-82e2-659889fe29ad admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:24.259842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:24.271549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4811af18-6f79-4b09-b876-05bc943ad1fd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1c5e4469a44446bbb87743271fb4b12a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:24.272255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4811af18-6f79-4b09-b876-05bc943ad1fd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1c5e4469a44446bbb87743271fb4b12a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:24.284969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4811af18-6f79-4b09-b876-05bc943ad1fd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1c5e4469a44446bbb87743271fb4b12a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:24.285920 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4811af18-6f79-4b09-b876-05bc943ad1fd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1c5e4469a44446bbb87743271fb4b12a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:24.295327 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:24.303515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4811af18-6f79-4b09-b876-05bc943ad1fd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1c5e4469a44446bbb87743271fb4b12a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:24.321300 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4c5fa3dd-269c-496f-82e2-659889fe29ad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a4dca1b96a7541698845aa6c79c71367'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:24.322389 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4c5fa3dd-269c-496f-82e2-659889fe29ad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a4dca1b96a7541698845aa6c79c71367'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:24.322571 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 955/1896] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:24 2022] DELETE /identity/v3/users/1c5e4469a44446bbb87743271fb4b12a => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:24.327678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6320c6eb-b277-4100-9538-9d21bebdf712 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:24.327678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6320c6eb-b277-4100-9538-9d21bebdf712 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:24.327909 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6320c6eb-b277-4100-9538-9d21bebdf712 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:24.333543 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4c5fa3dd-269c-496f-82e2-659889fe29ad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a4dca1b96a7541698845aa6c79c71367'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:24.334748 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4c5fa3dd-269c-496f-82e2-659889fe29ad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a4dca1b96a7541698845aa6c79c71367'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:24.346507 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4c5fa3dd-269c-496f-82e2-659889fe29ad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a4dca1b96a7541698845aa6c79c71367'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:24.349948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-6320c6eb-b277-4100-9538-9d21bebdf712 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:24.351115 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6320c6eb-b277-4100-9538-9d21bebdf712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:24.386838 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 942/1897] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:24 2022] DELETE /identity/v3/users/a4dca1b96a7541698845aa6c79c71367 => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:24.387801 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5fa3dd-269c-496f-82e2-659889fe29ad admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:24.389848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4c5fa3dd-269c-496f-82e2-659889fe29ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:24.430002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 956/1898] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:24.430002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305d3592-1258-4fe3-953c-4b26ef37271d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:24.430002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305d3592-1258-4fe3-953c-4b26ef37271d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:24.430002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305d3592-1258-4fe3-953c-4b26ef37271d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:24.448245 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-305d3592-1258-4fe3-953c-4b26ef37271d None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:24.449516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-305d3592-1258-4fe3-953c-4b26ef37271d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:24.465283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5fa3dd-269c-496f-82e2-659889fe29ad admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:24.469895 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18746aa-e7c1-4bdf-bd62-a5f829ab523d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:24.470519 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18746aa-e7c1-4bdf-bd62-a5f829ab523d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:24.470801 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18746aa-e7c1-4bdf-bd62-a5f829ab523d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:24.471104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18746aa-e7c1-4bdf-bd62-a5f829ab523d admin admin] PATH_INFO: `/v3/users/d062f99445b64b8a8451ee9e61e68d05` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:24.472236 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18746aa-e7c1-4bdf-bd62-a5f829ab523d admin admin] RBAC: Authorizing `identity:delete_user(user_id=d062f99445b64b8a8451ee9e61e68d05)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:24.474152 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18746aa-e7c1-4bdf-bd62-a5f829ab523d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:24.516189 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 957/1899] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:24.517152 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305d3592-1258-4fe3-953c-4b26ef37271d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:24.518413 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-305d3592-1258-4fe3-953c-4b26ef37271d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:24.528267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c18746aa-e7c1-4bdf-bd62-a5f829ab523d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd062f99445b64b8a8451ee9e61e68d05'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:24.529276 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c18746aa-e7c1-4bdf-bd62-a5f829ab523d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd062f99445b64b8a8451ee9e61e68d05'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:24.538580 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c18746aa-e7c1-4bdf-bd62-a5f829ab523d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd062f99445b64b8a8451ee9e61e68d05'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:24.539406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c18746aa-e7c1-4bdf-bd62-a5f829ab523d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd062f99445b64b8a8451ee9e61e68d05'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:24.546831 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c18746aa-e7c1-4bdf-bd62-a5f829ab523d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd062f99445b64b8a8451ee9e61e68d05'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:24.570382 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 943/1900] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:24 2022] DELETE /identity/v3/users/d062f99445b64b8a8451ee9e61e68d05 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:24.571420 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18746aa-e7c1-4bdf-bd62-a5f829ab523d admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:24.574385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c18746aa-e7c1-4bdf-bd62-a5f829ab523d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:24.586675 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305d3592-1258-4fe3-953c-4b26ef37271d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:24.592198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:24.612782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:24.644994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18746aa-e7c1-4bdf-bd62-a5f829ab523d admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:24.648531 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91fbfb3-738e-4120-8988-aef333a464b2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:24.648928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a21a37f-acf4-42c1-966c-1d8332a6f3b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:24.649699 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91fbfb3-738e-4120-8988-aef333a464b2 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:24.649699 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91fbfb3-738e-4120-8988-aef333a464b2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:24.649907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a21a37f-acf4-42c1-966c-1d8332a6f3b0 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:24.649979 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91fbfb3-738e-4120-8988-aef333a464b2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:24.650047 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a21a37f-acf4-42c1-966c-1d8332a6f3b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:24.650047 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a21a37f-acf4-42c1-966c-1d8332a6f3b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:24.651561 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b91fbfb3-738e-4120-8988-aef333a464b2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:24.651735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3a21a37f-acf4-42c1-966c-1d8332a6f3b0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:24.738359 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91fbfb3-738e-4120-8988-aef333a464b2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:24.740622 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91fbfb3-738e-4120-8988-aef333a464b2 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:24.752213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a21a37f-acf4-42c1-966c-1d8332a6f3b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:24.753694 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a21a37f-acf4-42c1-966c-1d8332a6f3b0 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:24.769019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 944/1901] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:24.770302 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f6471c-c08b-4061-9d57-180cedfc19e1 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:24.770986 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f6471c-c08b-4061-9d57-180cedfc19e1 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:24.771378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f6471c-c08b-4061-9d57-180cedfc19e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:24.778976 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 958/1902] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:24.795825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c2f6471c-c08b-4061-9d57-180cedfc19e1 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:24.797729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c2f6471c-c08b-4061-9d57-180cedfc19e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:24.824737 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 945/1903] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:24.841820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91fbfb3-738e-4120-8988-aef333a464b2 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:24.854238 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91fbfb3-738e-4120-8988-aef333a464b2 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:24.859465 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1751f74-462b-44ac-8467-0c9fea8c0a6b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:24.860331 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1751f74-462b-44ac-8467-0c9fea8c0a6b service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:24.860465 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1751f74-462b-44ac-8467-0c9fea8c0a6b service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:24.860708 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1751f74-462b-44ac-8467-0c9fea8c0a6b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:24.873903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1751f74-462b-44ac-8467-0c9fea8c0a6b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:24.875752 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1751f74-462b-44ac-8467-0c9fea8c0a6b service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:24.889652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 959/1904] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:19:25.129052 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f6471c-c08b-4061-9d57-180cedfc19e1 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:25.132249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c2f6471c-c08b-4061-9d57-180cedfc19e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:25.145049 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f6471c-c08b-4061-9d57-180cedfc19e1 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:25.152723 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb3e58f-501f-4014-b669-732c0cbd6cdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:25.153406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb3e58f-501f-4014-b669-732c0cbd6cdc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:25.153752 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb3e58f-501f-4014-b669-732c0cbd6cdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:25.154111 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb3e58f-501f-4014-b669-732c0cbd6cdc admin admin] PATH_INFO: `/v3/users/1b1c2bb130294b969d21071c9b93ea09` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:25.155881 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb3e58f-501f-4014-b669-732c0cbd6cdc admin admin] RBAC: Authorizing `identity:delete_user(user_id=1b1c2bb130294b969d21071c9b93ea09)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:25.158095 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb3e58f-501f-4014-b669-732c0cbd6cdc admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:25.220651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1751f74-462b-44ac-8467-0c9fea8c0a6b service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:25.222060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d1751f74-462b-44ac-8467-0c9fea8c0a6b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:25.226501 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1eb3e58f-501f-4014-b669-732c0cbd6cdc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1b1c2bb130294b969d21071c9b93ea09'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:25.227262 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1eb3e58f-501f-4014-b669-732c0cbd6cdc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1b1c2bb130294b969d21071c9b93ea09'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:25.236632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1eb3e58f-501f-4014-b669-732c0cbd6cdc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1b1c2bb130294b969d21071c9b93ea09'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:25.238016 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1eb3e58f-501f-4014-b669-732c0cbd6cdc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1b1c2bb130294b969d21071c9b93ea09'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:25.244951 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1751f74-462b-44ac-8467-0c9fea8c0a6b service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:25.253065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f488de6e-b909-4dcc-ba9e-f40f1b7cd1eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:25.254063 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f488de6e-b909-4dcc-ba9e-f40f1b7cd1eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:25.254461 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f488de6e-b909-4dcc-ba9e-f40f1b7cd1eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:25.254811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f488de6e-b909-4dcc-ba9e-f40f1b7cd1eb admin admin] PATH_INFO: `/v3/projects/e026e246277240e5b2606cd383148a5b` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:25.256221 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f488de6e-b909-4dcc-ba9e-f40f1b7cd1eb admin admin] RBAC: Authorizing `identity:delete_project(project_id=e026e246277240e5b2606cd383148a5b)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:25.259152 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f488de6e-b909-4dcc-ba9e-f40f1b7cd1eb admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:25.259619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1eb3e58f-501f-4014-b669-732c0cbd6cdc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1b1c2bb130294b969d21071c9b93ea09'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:25.300812 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 946/1905] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:25 2022] DELETE /identity/v3/users/1b1c2bb130294b969d21071c9b93ea09 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:25.304940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837d00cf-a6cc-43e2-986b-fcc2dbfec331 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:25.305227 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837d00cf-a6cc-43e2-986b-fcc2dbfec331 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:25.305504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837d00cf-a6cc-43e2-986b-fcc2dbfec331 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:25.329523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f488de6e-b909-4dcc-ba9e-f40f1b7cd1eb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e026e246277240e5b2606cd383148a5b', 'request_id': 'req-f488de6e-b909-4dcc-ba9e-f40f1b7cd1eb'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:25.337769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f488de6e-b909-4dcc-ba9e-f40f1b7cd1eb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e026e246277240e5b2606cd383148a5b', 'request_id': 'req-f488de6e-b909-4dcc-ba9e-f40f1b7cd1eb'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:25.338138 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-837d00cf-a6cc-43e2-986b-fcc2dbfec331 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:25.340146 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-837d00cf-a6cc-43e2-986b-fcc2dbfec331 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:25.345060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f488de6e-b909-4dcc-ba9e-f40f1b7cd1eb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e026e246277240e5b2606cd383148a5b', 'request_id': 'req-f488de6e-b909-4dcc-ba9e-f40f1b7cd1eb'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:25.365187 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f488de6e-b909-4dcc-ba9e-f40f1b7cd1eb admin admin] The token cache is being invalidate because project e026e246277240e5b2606cd383148a5b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:25.365488 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f488de6e-b909-4dcc-ba9e-f40f1b7cd1eb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:25.366155 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f488de6e-b909-4dcc-ba9e-f40f1b7cd1eb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:25.375109 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 960/1906] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:25 2022] DELETE /identity/v3/projects/e026e246277240e5b2606cd383148a5b => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:19:25.376210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f488de6e-b909-4dcc-ba9e-f40f1b7cd1eb admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:25.377590 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f488de6e-b909-4dcc-ba9e-f40f1b7cd1eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:25.418004 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:25.450803 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 947/1907] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:19:25.457997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:25.479827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837d00cf-a6cc-43e2-986b-fcc2dbfec331 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:25.482963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-837d00cf-a6cc-43e2-986b-fcc2dbfec331 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:25.523799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f488de6e-b909-4dcc-ba9e-f40f1b7cd1eb admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:25.536119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b516f4-fe9c-4309-8d50-558c26bcb6af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:25.537309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b516f4-fe9c-4309-8d50-558c26bcb6af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:25.537590 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b516f4-fe9c-4309-8d50-558c26bcb6af admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:25.539157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b516f4-fe9c-4309-8d50-558c26bcb6af admin admin] PATH_INFO: `/v3/users/2cc353987db54592ae437db64595db52` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:25.540530 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b516f4-fe9c-4309-8d50-558c26bcb6af admin admin] RBAC: Authorizing `identity:delete_user(user_id=2cc353987db54592ae437db64595db52)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:25.542574 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b516f4-fe9c-4309-8d50-558c26bcb6af admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:25.564252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837d00cf-a6cc-43e2-986b-fcc2dbfec331 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:25.574478 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bb7465-8b0d-473e-8bf5-12f9adbd8d19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:25.575346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bb7465-8b0d-473e-8bf5-12f9adbd8d19 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:25.575635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bb7465-8b0d-473e-8bf5-12f9adbd8d19 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:25.575903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bb7465-8b0d-473e-8bf5-12f9adbd8d19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:25.588531 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bb7465-8b0d-473e-8bf5-12f9adbd8d19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:25.590025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bb7465-8b0d-473e-8bf5-12f9adbd8d19 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:25.605463 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-57b516f4-fe9c-4309-8d50-558c26bcb6af admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2cc353987db54592ae437db64595db52'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:25.606318 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-57b516f4-fe9c-4309-8d50-558c26bcb6af admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2cc353987db54592ae437db64595db52'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:25.617622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 948/1908] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:25.619276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-57b516f4-fe9c-4309-8d50-558c26bcb6af admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2cc353987db54592ae437db64595db52'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:25.621579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-57b516f4-fe9c-4309-8d50-558c26bcb6af admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2cc353987db54592ae437db64595db52'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:25.649313 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-57b516f4-fe9c-4309-8d50-558c26bcb6af admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2cc353987db54592ae437db64595db52'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:25.670347 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 961/1909] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:25 2022] DELETE /identity/v3/users/2cc353987db54592ae437db64595db52 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:25.784340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bb7465-8b0d-473e-8bf5-12f9adbd8d19 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:25.789008 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-30bb7465-8b0d-473e-8bf5-12f9adbd8d19 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:25.813563 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bb7465-8b0d-473e-8bf5-12f9adbd8d19 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:25.998878 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2356107c-5062-422c-b202-08018e7c3ab1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:26.000899 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2356107c-5062-422c-b202-08018e7c3ab1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:26.001801 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2356107c-5062-422c-b202-08018e7c3ab1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:26.002224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2356107c-5062-422c-b202-08018e7c3ab1 admin admin] PATH_INFO: `/v3/projects/f0c6baba8aa248fdbaa02452ef21e987` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:26.010162 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2356107c-5062-422c-b202-08018e7c3ab1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f0c6baba8aa248fdbaa02452ef21e987)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:26.023498 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2356107c-5062-422c-b202-08018e7c3ab1 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:26.064134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3a06e2-11c1-4016-8fd6-08542330364a None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:26.064868 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3a06e2-11c1-4016-8fd6-08542330364a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:26.065372 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3a06e2-11c1-4016-8fd6-08542330364a None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:26.067348 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 962/1910] 10.222.0.15 () {54 vars in 921 bytes} [Tue Sep 13 01:19:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:19:26.090837 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26778aaa-2eb9-4dec-abda-14a31d9a6415 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:26.091211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26778aaa-2eb9-4dec-abda-14a31d9a6415 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:26.091587 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26778aaa-2eb9-4dec-abda-14a31d9a6415 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:26.127926 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-26778aaa-2eb9-4dec-abda-14a31d9a6415 None None] MFA Rules not processed for user `2f2da4d2d3604d16b58dc25b8c17c49a`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:26.129277 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-26778aaa-2eb9-4dec-abda-14a31d9a6415 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:26.178100 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2356107c-5062-422c-b202-08018e7c3ab1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f0c6baba8aa248fdbaa02452ef21e987', 'request_id': 'req-2356107c-5062-422c-b202-08018e7c3ab1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.191361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2356107c-5062-422c-b202-08018e7c3ab1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f0c6baba8aa248fdbaa02452ef21e987', 'request_id': 'req-2356107c-5062-422c-b202-08018e7c3ab1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.233287 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2356107c-5062-422c-b202-08018e7c3ab1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f0c6baba8aa248fdbaa02452ef21e987', 'request_id': 'req-2356107c-5062-422c-b202-08018e7c3ab1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.264596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 963/1911] 10.222.0.15 () {58 vars in 1018 bytes} [Tue Sep 13 01:19:26 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:19:26.265774 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26778aaa-2eb9-4dec-abda-14a31d9a6415 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:26.268180 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-26778aaa-2eb9-4dec-abda-14a31d9a6415 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:26.278998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2356107c-5062-422c-b202-08018e7c3ab1 admin admin] The token cache is being invalidate because project f0c6baba8aa248fdbaa02452ef21e987 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:26.279403 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2356107c-5062-422c-b202-08018e7c3ab1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.281394 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2356107c-5062-422c-b202-08018e7c3ab1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.283090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 949/1912] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:25 2022] DELETE /identity/v3/projects/f0c6baba8aa248fdbaa02452ef21e987 => generated 0 bytes in 504 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:26.288511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2356107c-5062-422c-b202-08018e7c3ab1 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:26.290313 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2356107c-5062-422c-b202-08018e7c3ab1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:26.335688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:26.349462 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:26.369905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26778aaa-2eb9-4dec-abda-14a31d9a6415 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:26.378925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c18a8d-f331-454e-bf92-5328b26d0ae0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:26.379646 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c18a8d-f331-454e-bf92-5328b26d0ae0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:26.380074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c18a8d-f331-454e-bf92-5328b26d0ae0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:26.380358 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c18a8d-f331-454e-bf92-5328b26d0ae0 admin admin] PATH_INFO: `/v3/projects/8b7ef23e634648bcb97f0ce19114b8f4` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:26.382418 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c18a8d-f331-454e-bf92-5328b26d0ae0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8b7ef23e634648bcb97f0ce19114b8f4)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:26.384430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c18a8d-f331-454e-bf92-5328b26d0ae0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:26.458399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-14c18a8d-f331-454e-bf92-5328b26d0ae0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8b7ef23e634648bcb97f0ce19114b8f4', 'request_id': 'req-14c18a8d-f331-454e-bf92-5328b26d0ae0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.467592 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-14c18a8d-f331-454e-bf92-5328b26d0ae0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8b7ef23e634648bcb97f0ce19114b8f4', 'request_id': 'req-14c18a8d-f331-454e-bf92-5328b26d0ae0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.476142 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-14c18a8d-f331-454e-bf92-5328b26d0ae0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8b7ef23e634648bcb97f0ce19114b8f4', 'request_id': 'req-14c18a8d-f331-454e-bf92-5328b26d0ae0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.496727 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-14c18a8d-f331-454e-bf92-5328b26d0ae0 admin admin] The token cache is being invalidate because project 8b7ef23e634648bcb97f0ce19114b8f4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:26.496727 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-14c18a8d-f331-454e-bf92-5328b26d0ae0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.497004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-14c18a8d-f331-454e-bf92-5328b26d0ae0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.502739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 964/1913] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:26 2022] DELETE /identity/v3/projects/8b7ef23e634648bcb97f0ce19114b8f4 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:26.504510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c18a8d-f331-454e-bf92-5328b26d0ae0 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:26.506217 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-14c18a8d-f331-454e-bf92-5328b26d0ae0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:26.564251 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2356107c-5062-422c-b202-08018e7c3ab1 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:26.631427 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c18a8d-f331-454e-bf92-5328b26d0ae0 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:26.637116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36eb9c69-d495-4568-8f45-0fe4b4a66509 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:26.637772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eb9c69-d495-4568-8f45-0fe4b4a66509 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:26.638057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eb9c69-d495-4568-8f45-0fe4b4a66509 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:26.638327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eb9c69-d495-4568-8f45-0fe4b4a66509 admin admin] PATH_INFO: `/v3/projects/041f608de8e34c8bac1cede0693f7112` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:26.639692 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36eb9c69-d495-4568-8f45-0fe4b4a66509 admin admin] RBAC: Authorizing `identity:delete_project(project_id=041f608de8e34c8bac1cede0693f7112)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:26.642366 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36eb9c69-d495-4568-8f45-0fe4b4a66509 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:26.699097 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac08a373-74d4-4ced-8617-ab74dcf7506b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:26.699784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac08a373-74d4-4ced-8617-ab74dcf7506b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:26.700063 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac08a373-74d4-4ced-8617-ab74dcf7506b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:26.700341 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac08a373-74d4-4ced-8617-ab74dcf7506b admin admin] PATH_INFO: `/v3/projects/c74ba6325b1e4b49bb9531d3aac902ea` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:26.701756 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac08a373-74d4-4ced-8617-ab74dcf7506b admin admin] RBAC: Authorizing `identity:delete_project(project_id=c74ba6325b1e4b49bb9531d3aac902ea)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:26.703831 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac08a373-74d4-4ced-8617-ab74dcf7506b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:26.722598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-36eb9c69-d495-4568-8f45-0fe4b4a66509 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '041f608de8e34c8bac1cede0693f7112', 'request_id': 'req-36eb9c69-d495-4568-8f45-0fe4b4a66509'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.733412 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-36eb9c69-d495-4568-8f45-0fe4b4a66509 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '041f608de8e34c8bac1cede0693f7112', 'request_id': 'req-36eb9c69-d495-4568-8f45-0fe4b4a66509'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.745428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-36eb9c69-d495-4568-8f45-0fe4b4a66509 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '041f608de8e34c8bac1cede0693f7112', 'request_id': 'req-36eb9c69-d495-4568-8f45-0fe4b4a66509'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.763652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:26.770109 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-36eb9c69-d495-4568-8f45-0fe4b4a66509 admin admin] The token cache is being invalidate because project 041f608de8e34c8bac1cede0693f7112 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:26.771014 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-36eb9c69-d495-4568-8f45-0fe4b4a66509 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.771835 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ac08a373-74d4-4ced-8617-ab74dcf7506b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c74ba6325b1e4b49bb9531d3aac902ea', 'request_id': 'req-ac08a373-74d4-4ced-8617-ab74dcf7506b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.772078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-36eb9c69-d495-4568-8f45-0fe4b4a66509 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.774007 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 965/1914] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:26 2022] DELETE /identity/v3/projects/041f608de8e34c8bac1cede0693f7112 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:26.781912 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36eb9c69-d495-4568-8f45-0fe4b4a66509 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:26.784235 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-36eb9c69-d495-4568-8f45-0fe4b4a66509 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:26.788931 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ac08a373-74d4-4ced-8617-ab74dcf7506b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c74ba6325b1e4b49bb9531d3aac902ea', 'request_id': 'req-ac08a373-74d4-4ced-8617-ab74dcf7506b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.800603 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ac08a373-74d4-4ced-8617-ab74dcf7506b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c74ba6325b1e4b49bb9531d3aac902ea', 'request_id': 'req-ac08a373-74d4-4ced-8617-ab74dcf7506b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.822246 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ac08a373-74d4-4ced-8617-ab74dcf7506b admin admin] The token cache is being invalidate because project c74ba6325b1e4b49bb9531d3aac902ea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:26.822815 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ac08a373-74d4-4ced-8617-ab74dcf7506b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.823821 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ac08a373-74d4-4ced-8617-ab74dcf7506b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.826471 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 950/1915] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:26 2022] DELETE /identity/v3/projects/c74ba6325b1e4b49bb9531d3aac902ea => generated 0 bytes in 543 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:26.827501 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac08a373-74d4-4ced-8617-ab74dcf7506b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:26.830520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ac08a373-74d4-4ced-8617-ab74dcf7506b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:26.892289 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36eb9c69-d495-4568-8f45-0fe4b4a66509 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:26.908786 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:26.926643 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac08a373-74d4-4ced-8617-ab74dcf7506b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:26.934575 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6c5e61-7440-4caa-9c37-af31d07f12aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:26.935735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6c5e61-7440-4caa-9c37-af31d07f12aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:26.935735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6c5e61-7440-4caa-9c37-af31d07f12aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:26.936301 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6c5e61-7440-4caa-9c37-af31d07f12aa admin admin] PATH_INFO: `/v3/users/56e941e0cd314f8fa67896e4801500ad` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:26.937469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6c5e61-7440-4caa-9c37-af31d07f12aa admin admin] RBAC: Authorizing `identity:delete_user(user_id=56e941e0cd314f8fa67896e4801500ad)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:26.939909 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6c5e61-7440-4caa-9c37-af31d07f12aa admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:26.962266 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a3b243-daaa-478d-a0aa-57b8c0273cf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:26.963102 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a3b243-daaa-478d-a0aa-57b8c0273cf2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:26.963380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a3b243-daaa-478d-a0aa-57b8c0273cf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:26.963652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a3b243-daaa-478d-a0aa-57b8c0273cf2 admin admin] PATH_INFO: `/v3/users/b1ac05cfe1e74a9da9201644dfc14f22` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:26.965575 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a3b243-daaa-478d-a0aa-57b8c0273cf2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b1ac05cfe1e74a9da9201644dfc14f22)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:26.969022 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a3b243-daaa-478d-a0aa-57b8c0273cf2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:26.997798 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1b6c5e61-7440-4caa-9c37-af31d07f12aa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '56e941e0cd314f8fa67896e4801500ad'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:26.998704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1b6c5e61-7440-4caa-9c37-af31d07f12aa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '56e941e0cd314f8fa67896e4801500ad'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:27.011688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1b6c5e61-7440-4caa-9c37-af31d07f12aa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '56e941e0cd314f8fa67896e4801500ad'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:27.012434 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1b6c5e61-7440-4caa-9c37-af31d07f12aa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '56e941e0cd314f8fa67896e4801500ad'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:27.021994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1b6c5e61-7440-4caa-9c37-af31d07f12aa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '56e941e0cd314f8fa67896e4801500ad'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:27.032435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-42a3b243-daaa-478d-a0aa-57b8c0273cf2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b1ac05cfe1e74a9da9201644dfc14f22'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:27.033321 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-42a3b243-daaa-478d-a0aa-57b8c0273cf2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b1ac05cfe1e74a9da9201644dfc14f22'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:27.042900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 951/1916] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:26 2022] DELETE /identity/v3/users/56e941e0cd314f8fa67896e4801500ad => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:27.043112 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6c5e61-7440-4caa-9c37-af31d07f12aa admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:27.044730 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1b6c5e61-7440-4caa-9c37-af31d07f12aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:27.053438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-42a3b243-daaa-478d-a0aa-57b8c0273cf2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b1ac05cfe1e74a9da9201644dfc14f22'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:27.054380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-42a3b243-daaa-478d-a0aa-57b8c0273cf2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b1ac05cfe1e74a9da9201644dfc14f22'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:27.076927 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-42a3b243-daaa-478d-a0aa-57b8c0273cf2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b1ac05cfe1e74a9da9201644dfc14f22'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:27.122686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 966/1917] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:26 2022] DELETE /identity/v3/users/b1ac05cfe1e74a9da9201644dfc14f22 => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:27.124756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a3b243-daaa-478d-a0aa-57b8c0273cf2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:27.129554 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-42a3b243-daaa-478d-a0aa-57b8c0273cf2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:27.149815 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6c5e61-7440-4caa-9c37-af31d07f12aa admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:27.210827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:27.257147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578fd8e9-8d79-4a3b-98c4-cab6acc64a4b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:27.257715 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578fd8e9-8d79-4a3b-98c4-cab6acc64a4b service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:27.258014 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578fd8e9-8d79-4a3b-98c4-cab6acc64a4b service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:27.258734 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578fd8e9-8d79-4a3b-98c4-cab6acc64a4b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:27.261967 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-578fd8e9-8d79-4a3b-98c4-cab6acc64a4b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:27.275263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:27.276703 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a3b243-daaa-478d-a0aa-57b8c0273cf2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:27.280366 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578fd8e9-8d79-4a3b-98c4-cab6acc64a4b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:27.283246 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578fd8e9-8d79-4a3b-98c4-cab6acc64a4b service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:27.285782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f006b6b-55de-4ea4-9709-60d75afb9cc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:27.287211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f006b6b-55de-4ea4-9709-60d75afb9cc0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:27.287646 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f006b6b-55de-4ea4-9709-60d75afb9cc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:27.288039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f006b6b-55de-4ea4-9709-60d75afb9cc0 admin admin] PATH_INFO: `/v3/users/acb01352910e4d228874d13e06d66ef1` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:27.290127 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f006b6b-55de-4ea4-9709-60d75afb9cc0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=acb01352910e4d228874d13e06d66ef1)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:27.295273 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f006b6b-55de-4ea4-9709-60d75afb9cc0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:27.305126 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 952/1918] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:27 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 262 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:27.306227 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578fd8e9-8d79-4a3b-98c4-cab6acc64a4b service nova] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:27.308301 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-578fd8e9-8d79-4a3b-98c4-cab6acc64a4b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:27.325493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578fd8e9-8d79-4a3b-98c4-cab6acc64a4b service nova] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:27.337592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d02dc2-6f82-4edc-a498-f9e0a0c9697a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:27.338603 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d02dc2-6f82-4edc-a498-f9e0a0c9697a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:27.338986 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d02dc2-6f82-4edc-a498-f9e0a0c9697a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:27.339293 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d02dc2-6f82-4edc-a498-f9e0a0c9697a admin admin] PATH_INFO: `/v3/users/99327d68a5f845b0a44224911ecb7f3b` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:27.340804 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d02dc2-6f82-4edc-a498-f9e0a0c9697a admin admin] RBAC: Authorizing `identity:delete_user(user_id=99327d68a5f845b0a44224911ecb7f3b)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:27.343818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d02dc2-6f82-4edc-a498-f9e0a0c9697a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:27.379615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6f006b6b-55de-4ea4-9709-60d75afb9cc0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'acb01352910e4d228874d13e06d66ef1'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:27.380638 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6f006b6b-55de-4ea4-9709-60d75afb9cc0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'acb01352910e4d228874d13e06d66ef1'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:27.391537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6f006b6b-55de-4ea4-9709-60d75afb9cc0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'acb01352910e4d228874d13e06d66ef1'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:27.392666 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6f006b6b-55de-4ea4-9709-60d75afb9cc0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'acb01352910e4d228874d13e06d66ef1'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:27.422978 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6f006b6b-55de-4ea4-9709-60d75afb9cc0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'acb01352910e4d228874d13e06d66ef1'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:27.423228 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-45d02dc2-6f82-4edc-a498-f9e0a0c9697a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '99327d68a5f845b0a44224911ecb7f3b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:27.424025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-45d02dc2-6f82-4edc-a498-f9e0a0c9697a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '99327d68a5f845b0a44224911ecb7f3b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:27.436485 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-45d02dc2-6f82-4edc-a498-f9e0a0c9697a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '99327d68a5f845b0a44224911ecb7f3b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:27.439184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-45d02dc2-6f82-4edc-a498-f9e0a0c9697a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '99327d68a5f845b0a44224911ecb7f3b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:27.449313 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-45d02dc2-6f82-4edc-a498-f9e0a0c9697a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '99327d68a5f845b0a44224911ecb7f3b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:27.451123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 967/1919] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:27 2022] DELETE /identity/v3/users/acb01352910e4d228874d13e06d66ef1 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:27.477723 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 953/1920] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:27 2022] DELETE /identity/v3/users/99327d68a5f845b0a44224911ecb7f3b => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:27.860302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f006b6b-55de-4ea4-9709-60d75afb9cc0 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:27.861952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6f006b6b-55de-4ea4-9709-60d75afb9cc0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:28.033942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f006b6b-55de-4ea4-9709-60d75afb9cc0 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:28.054595 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d02dc2-6f82-4edc-a498-f9e0a0c9697a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:28.056362 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-45d02dc2-6f82-4edc-a498-f9e0a0c9697a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:28.169939 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:28.188247 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4c2972-3cb0-459b-9c80-a8e866e4c9b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:28.189804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4c2972-3cb0-459b-9c80-a8e866e4c9b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:28.190363 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4c2972-3cb0-459b-9c80-a8e866e4c9b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:28.190842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4c2972-3cb0-459b-9c80-a8e866e4c9b8 admin admin] PATH_INFO: `/v3/projects/bba09ef23dae4910a89daa82bb8f447a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:28.194806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4c2972-3cb0-459b-9c80-a8e866e4c9b8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bba09ef23dae4910a89daa82bb8f447a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:28.197454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4c2972-3cb0-459b-9c80-a8e866e4c9b8 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:28.268175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d02dc2-6f82-4edc-a498-f9e0a0c9697a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:28.285218 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528184eb-c851-4a51-95ea-d331314e4f23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:28.288433 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528184eb-c851-4a51-95ea-d331314e4f23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:28.289997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528184eb-c851-4a51-95ea-d331314e4f23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:28.290294 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528184eb-c851-4a51-95ea-d331314e4f23 admin admin] PATH_INFO: `/v3/projects/cb38177fb9b445859cd177334a1d2c7a` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:28.295809 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528184eb-c851-4a51-95ea-d331314e4f23 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb38177fb9b445859cd177334a1d2c7a)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:28.303768 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4e4c2972-3cb0-459b-9c80-a8e866e4c9b8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bba09ef23dae4910a89daa82bb8f447a', 'request_id': 'req-4e4c2972-3cb0-459b-9c80-a8e866e4c9b8'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.307149 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528184eb-c851-4a51-95ea-d331314e4f23 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:28.315430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4e4c2972-3cb0-459b-9c80-a8e866e4c9b8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bba09ef23dae4910a89daa82bb8f447a', 'request_id': 'req-4e4c2972-3cb0-459b-9c80-a8e866e4c9b8'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.325008 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4e4c2972-3cb0-459b-9c80-a8e866e4c9b8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bba09ef23dae4910a89daa82bb8f447a', 'request_id': 'req-4e4c2972-3cb0-459b-9c80-a8e866e4c9b8'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.348297 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4e4c2972-3cb0-459b-9c80-a8e866e4c9b8 admin admin] The token cache is being invalidate because project bba09ef23dae4910a89daa82bb8f447a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:28.349179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4e4c2972-3cb0-459b-9c80-a8e866e4c9b8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.350060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-4e4c2972-3cb0-459b-9c80-a8e866e4c9b8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.352391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 968/1921] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:27 2022] DELETE /identity/v3/projects/bba09ef23dae4910a89daa82bb8f447a => generated 0 bytes in 493 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:28.353557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4c2972-3cb0-459b-9c80-a8e866e4c9b8 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:28.354969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4e4c2972-3cb0-459b-9c80-a8e866e4c9b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:28.427342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-528184eb-c851-4a51-95ea-d331314e4f23 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb38177fb9b445859cd177334a1d2c7a', 'request_id': 'req-528184eb-c851-4a51-95ea-d331314e4f23'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.436535 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-528184eb-c851-4a51-95ea-d331314e4f23 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb38177fb9b445859cd177334a1d2c7a', 'request_id': 'req-528184eb-c851-4a51-95ea-d331314e4f23'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.445642 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-528184eb-c851-4a51-95ea-d331314e4f23 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb38177fb9b445859cd177334a1d2c7a', 'request_id': 'req-528184eb-c851-4a51-95ea-d331314e4f23'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.456829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:28.464600 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-528184eb-c851-4a51-95ea-d331314e4f23 admin admin] The token cache is being invalidate because project cb38177fb9b445859cd177334a1d2c7a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:28.464906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-528184eb-c851-4a51-95ea-d331314e4f23 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.466607 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-528184eb-c851-4a51-95ea-d331314e4f23 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.468232 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 954/1922] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:28 2022] DELETE /identity/v3/projects/cb38177fb9b445859cd177334a1d2c7a => generated 0 bytes in 414 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:19:28.469804 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fef673c-9401-457d-9a8e-905bb8fa94c7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:28.470409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fef673c-9401-457d-9a8e-905bb8fa94c7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:28.471051 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fef673c-9401-457d-9a8e-905bb8fa94c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:28.475262 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4c2972-3cb0-459b-9c80-a8e866e4c9b8 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:28.491498 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6fef673c-9401-457d-9a8e-905bb8fa94c7 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:28.492709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6fef673c-9401-457d-9a8e-905bb8fa94c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:28.536354 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8094c4-e7d1-45c4-a362-535c6fb718be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:28.536969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8094c4-e7d1-45c4-a362-535c6fb718be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:28.537245 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8094c4-e7d1-45c4-a362-535c6fb718be admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:28.537566 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8094c4-e7d1-45c4-a362-535c6fb718be admin admin] PATH_INFO: `/v3/projects/6ecae603a45d42d4b787cbdccb4678d4` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:28.539416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8094c4-e7d1-45c4-a362-535c6fb718be admin admin] RBAC: Authorizing `identity:delete_project(project_id=6ecae603a45d42d4b787cbdccb4678d4)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:28.541871 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8094c4-e7d1-45c4-a362-535c6fb718be admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:28.557888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 955/1923] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:28.563058 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fef673c-9401-457d-9a8e-905bb8fa94c7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:28.565371 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6fef673c-9401-457d-9a8e-905bb8fa94c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:28.579345 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fef673c-9401-457d-9a8e-905bb8fa94c7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:28.580937 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:28.599701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6b8094c4-e7d1-45c4-a362-535c6fb718be admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6ecae603a45d42d4b787cbdccb4678d4', 'request_id': 'req-6b8094c4-e7d1-45c4-a362-535c6fb718be'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.606888 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6b8094c4-e7d1-45c4-a362-535c6fb718be admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6ecae603a45d42d4b787cbdccb4678d4', 'request_id': 'req-6b8094c4-e7d1-45c4-a362-535c6fb718be'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.615422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6b8094c4-e7d1-45c4-a362-535c6fb718be admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6ecae603a45d42d4b787cbdccb4678d4', 'request_id': 'req-6b8094c4-e7d1-45c4-a362-535c6fb718be'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.632457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6b8094c4-e7d1-45c4-a362-535c6fb718be admin admin] The token cache is being invalidate because project 6ecae603a45d42d4b787cbdccb4678d4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:28.633150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6b8094c4-e7d1-45c4-a362-535c6fb718be admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.633635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6b8094c4-e7d1-45c4-a362-535c6fb718be admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.636346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 969/1924] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:28 2022] DELETE /identity/v3/projects/6ecae603a45d42d4b787cbdccb4678d4 => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:28.637521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7488f2-63fa-4174-8990-b283b68293d4 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:28.637808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7488f2-63fa-4174-8990-b283b68293d4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:28.638075 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7488f2-63fa-4174-8990-b283b68293d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:28.644194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d840fa-e1d6-4af9-a336-a0210c0bce74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:28.644834 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d840fa-e1d6-4af9-a336-a0210c0bce74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:28.645174 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d840fa-e1d6-4af9-a336-a0210c0bce74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:28.645455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d840fa-e1d6-4af9-a336-a0210c0bce74 admin admin] PATH_INFO: `/v3/projects/e19cea5ab3524c1091f8c3e10f812086` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:28.646600 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d840fa-e1d6-4af9-a336-a0210c0bce74 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e19cea5ab3524c1091f8c3e10f812086)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:28.648764 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d840fa-e1d6-4af9-a336-a0210c0bce74 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:28.662845 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9a7488f2-63fa-4174-8990-b283b68293d4 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:28.664162 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9a7488f2-63fa-4174-8990-b283b68293d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:28.680172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 970/1925] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:28.680172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e958a69b-9539-471a-83d9-52043c11de58 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:28.680172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e958a69b-9539-471a-83d9-52043c11de58 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:28.680172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e958a69b-9539-471a-83d9-52043c11de58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:28.704853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e958a69b-9539-471a-83d9-52043c11de58 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:28.706323 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e958a69b-9539-471a-83d9-52043c11de58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:28.706700 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d8d840fa-e1d6-4af9-a336-a0210c0bce74 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e19cea5ab3524c1091f8c3e10f812086', 'request_id': 'req-d8d840fa-e1d6-4af9-a336-a0210c0bce74'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.712542 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d8d840fa-e1d6-4af9-a336-a0210c0bce74 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e19cea5ab3524c1091f8c3e10f812086', 'request_id': 'req-d8d840fa-e1d6-4af9-a336-a0210c0bce74'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.719813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d8d840fa-e1d6-4af9-a336-a0210c0bce74 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e19cea5ab3524c1091f8c3e10f812086', 'request_id': 'req-d8d840fa-e1d6-4af9-a336-a0210c0bce74'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.734024 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d8d840fa-e1d6-4af9-a336-a0210c0bce74 admin admin] The token cache is being invalidate because project e19cea5ab3524c1091f8c3e10f812086 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:28.734340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d8d840fa-e1d6-4af9-a336-a0210c0bce74 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.736147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d8d840fa-e1d6-4af9-a336-a0210c0bce74 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:28.737676 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 956/1926] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:28 2022] DELETE /identity/v3/projects/e19cea5ab3524c1091f8c3e10f812086 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:28.740158 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b8a376-6eeb-43bc-b646-2cde788866c9 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:28.741073 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b8a376-6eeb-43bc-b646-2cde788866c9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:28.741348 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b8a376-6eeb-43bc-b646-2cde788866c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:28.762239 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-78b8a376-6eeb-43bc-b646-2cde788866c9 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:28.763616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-78b8a376-6eeb-43bc-b646-2cde788866c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:28.776465 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 971/1927] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:28.777658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9bb56a-b65a-4033-91ef-2ec05daf4eef None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:28.777949 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9bb56a-b65a-4033-91ef-2ec05daf4eef None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:28.778261 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9bb56a-b65a-4033-91ef-2ec05daf4eef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:28.798757 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ef9bb56a-b65a-4033-91ef-2ec05daf4eef None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:28.799018 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:28.799999 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ef9bb56a-b65a-4033-91ef-2ec05daf4eef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:28.809193 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 972/1928] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:28.811380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e89843-0971-4884-98d3-442396dda7f6 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:28.811622 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e89843-0971-4884-98d3-442396dda7f6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:28.811889 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e89843-0971-4884-98d3-442396dda7f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:28.835703 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b2e89843-0971-4884-98d3-442396dda7f6 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:28.836949 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b2e89843-0971-4884-98d3-442396dda7f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:28.848319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 973/1929] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:28.849360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e89843-0971-4884-98d3-442396dda7f6 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:28.857205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 957/1930] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:28.858265 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b8a376-6eeb-43bc-b646-2cde788866c9 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:28.861682 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e89843-0971-4884-98d3-442396dda7f6 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:28.867781 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b428776-dd4e-4cdf-9ed6-392a541cca44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:28.868757 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b428776-dd4e-4cdf-9ed6-392a541cca44 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:28.869219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b428776-dd4e-4cdf-9ed6-392a541cca44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:28.869674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b428776-dd4e-4cdf-9ed6-392a541cca44 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:28.870518 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b428776-dd4e-4cdf-9ed6-392a541cca44 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:19:28.870825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b428776-dd4e-4cdf-9ed6-392a541cca44 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:28.872461 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b428776-dd4e-4cdf-9ed6-392a541cca44 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:28.879405 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b8a376-6eeb-43bc-b646-2cde788866c9 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:28.890207 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 974/1931] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:19:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:19:28.890207 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b0a0f3-d758-4768-8d99-823796115d65 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:28.890207 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b0a0f3-d758-4768-8d99-823796115d65 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:28.890207 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b0a0f3-d758-4768-8d99-823796115d65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:28.893901 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d575a29f-dead-4a47-a292-fd5dbd0ba2f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:28.894533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d575a29f-dead-4a47-a292-fd5dbd0ba2f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:28.895159 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d575a29f-dead-4a47-a292-fd5dbd0ba2f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:28.895159 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d575a29f-dead-4a47-a292-fd5dbd0ba2f4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:28.895827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d575a29f-dead-4a47-a292-fd5dbd0ba2f4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:19:28.896131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d575a29f-dead-4a47-a292-fd5dbd0ba2f4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:28.898032 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d575a29f-dead-4a47-a292-fd5dbd0ba2f4 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:28.912140 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 958/1932] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:19:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:19:28.913658 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760b227d-55c7-4c7a-aff7-547b28032210 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:28.913958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760b227d-55c7-4c7a-aff7-547b28032210 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:28.914254 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760b227d-55c7-4c7a-aff7-547b28032210 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:28.917315 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-53b0a0f3-d758-4768-8d99-823796115d65 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:28.919978 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-53b0a0f3-d758-4768-8d99-823796115d65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:28.932039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 975/1933] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:28.933158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b0a0f3-d758-4768-8d99-823796115d65 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:28.943415 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-760b227d-55c7-4c7a-aff7-547b28032210 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:28.944946 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-760b227d-55c7-4c7a-aff7-547b28032210 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:28.946130 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b0a0f3-d758-4768-8d99-823796115d65 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:28.951961 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c0e405-6cdb-443f-aca1-da3845bbb6a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:28.952682 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c0e405-6cdb-443f-aca1-da3845bbb6a7 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:28.953182 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c0e405-6cdb-443f-aca1-da3845bbb6a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:28.953528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c0e405-6cdb-443f-aca1-da3845bbb6a7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:28.954230 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c0e405-6cdb-443f-aca1-da3845bbb6a7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:28.956220 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c0e405-6cdb-443f-aca1-da3845bbb6a7 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:28.958753 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 959/1934] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:28.959971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760b227d-55c7-4c7a-aff7-547b28032210 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:28.972815 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760b227d-55c7-4c7a-aff7-547b28032210 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:28.985885 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22eb7b2d-d5d7-4638-b9c3-5254af3b9262 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:28.987168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22eb7b2d-d5d7-4638-b9c3-5254af3b9262 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:28.987458 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22eb7b2d-d5d7-4638-b9c3-5254af3b9262 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:28.987733 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22eb7b2d-d5d7-4638-b9c3-5254af3b9262 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:28.988481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22eb7b2d-d5d7-4638-b9c3-5254af3b9262 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:28.991189 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22eb7b2d-d5d7-4638-b9c3-5254af3b9262 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.003044 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 976/1935] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:28 2022] POST /identity/v3/projects => generated 381 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:29.007144 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c0e405-6cdb-443f-aca1-da3845bbb6a7 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.040369 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 960/1936] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:28 2022] POST /identity/v3/projects => generated 399 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:29.040369 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22eb7b2d-d5d7-4638-b9c3-5254af3b9262 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.081152 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c0e405-6cdb-443f-aca1-da3845bbb6a7 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.084930 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:29.108658 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22eb7b2d-d5d7-4638-b9c3-5254af3b9262 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.113394 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93b6b00-83c9-4707-89ca-69fc4f140ba3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.114129 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93b6b00-83c9-4707-89ca-69fc4f140ba3 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.114413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93b6b00-83c9-4707-89ca-69fc4f140ba3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.114828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93b6b00-83c9-4707-89ca-69fc4f140ba3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.115484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93b6b00-83c9-4707-89ca-69fc4f140ba3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:19:29.115787 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93b6b00-83c9-4707-89ca-69fc4f140ba3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.117709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93b6b00-83c9-4707-89ca-69fc4f140ba3 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.129763 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 961/1937] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:19:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:19:29.130685 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93b6b00-83c9-4707-89ca-69fc4f140ba3 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.140747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93b6b00-83c9-4707-89ca-69fc4f140ba3 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.144520 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e31edf-6435-40b2-a12f-3f2248c2b936 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.145145 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e31edf-6435-40b2-a12f-3f2248c2b936 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.145249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a042e11a-b51a-4171-8b7c-abb0eb4aaa58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.145483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e31edf-6435-40b2-a12f-3f2248c2b936 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.145764 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e31edf-6435-40b2-a12f-3f2248c2b936 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.145851 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a042e11a-b51a-4171-8b7c-abb0eb4aaa58 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.146087 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a042e11a-b51a-4171-8b7c-abb0eb4aaa58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.146357 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a042e11a-b51a-4171-8b7c-abb0eb4aaa58 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.146456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e31edf-6435-40b2-a12f-3f2248c2b936 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:19:29.146700 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e31edf-6435-40b2-a12f-3f2248c2b936 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.146996 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a042e11a-b51a-4171-8b7c-abb0eb4aaa58 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.148336 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e31edf-6435-40b2-a12f-3f2248c2b936 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.148867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a042e11a-b51a-4171-8b7c-abb0eb4aaa58 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.159879 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 977/1938] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:19:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:19:29.160662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e31edf-6435-40b2-a12f-3f2248c2b936 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.171340 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e31edf-6435-40b2-a12f-3f2248c2b936 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.176387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 962/1939] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:29 2022] POST /identity/v3/users => generated 291 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:29.176510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2a655f-7c63-44cb-beb6-653dab02f57d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.176872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a042e11a-b51a-4171-8b7c-abb0eb4aaa58 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.177078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2a655f-7c63-44cb-beb6-653dab02f57d admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.177362 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2a655f-7c63-44cb-beb6-653dab02f57d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.177676 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2a655f-7c63-44cb-beb6-653dab02f57d admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.178314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2a655f-7c63-44cb-beb6-653dab02f57d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.180187 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2a655f-7c63-44cb-beb6-653dab02f57d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.189625 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a042e11a-b51a-4171-8b7c-abb0eb4aaa58 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.196826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97797793-3a85-4ef8-ae0f-6e45d35e22a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.197462 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97797793-3a85-4ef8-ae0f-6e45d35e22a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.197799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97797793-3a85-4ef8-ae0f-6e45d35e22a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.198209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97797793-3a85-4ef8-ae0f-6e45d35e22a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.199457 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97797793-3a85-4ef8-ae0f-6e45d35e22a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.201438 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97797793-3a85-4ef8-ae0f-6e45d35e22a1 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.207387 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 978/1940] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:29 2022] POST /identity/v3/users => generated 300 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:29.208066 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2a655f-7c63-44cb-beb6-653dab02f57d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.218027 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2a655f-7c63-44cb-beb6-653dab02f57d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.222879 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4029dc-e136-4552-a9c4-9891447795d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.223629 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4029dc-e136-4552-a9c4-9891447795d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.223920 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4029dc-e136-4552-a9c4-9891447795d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.224217 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4029dc-e136-4552-a9c4-9891447795d6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.224891 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4029dc-e136-4552-a9c4-9891447795d6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.226986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4029dc-e136-4552-a9c4-9891447795d6 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.236820 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 963/1941] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:29 2022] POST /identity/v3/projects => generated 359 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:29.238068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97797793-3a85-4ef8-ae0f-6e45d35e22a1 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.267464 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 979/1942] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:29 2022] POST /identity/v3/projects => generated 387 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:29.267464 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4029dc-e136-4552-a9c4-9891447795d6 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.286845 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:29.314440 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97797793-3a85-4ef8-ae0f-6e45d35e22a1 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.342229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4029dc-e136-4552-a9c4-9891447795d6 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.351363 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdb189d-7c49-4b18-aa92-f53cfca8a939 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.352010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdb189d-7c49-4b18-aa92-f53cfca8a939 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.352284 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdb189d-7c49-4b18-aa92-f53cfca8a939 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.352553 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdb189d-7c49-4b18-aa92-f53cfca8a939 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.353221 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdb189d-7c49-4b18-aa92-f53cfca8a939 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.354791 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdb189d-7c49-4b18-aa92-f53cfca8a939 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.365751 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 980/1943] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:29 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:29.366584 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdb189d-7c49-4b18-aa92-f53cfca8a939 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.377010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdb189d-7c49-4b18-aa92-f53cfca8a939 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.381865 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5b71cc-9aee-4c64-9dc8-9cfa0bf76024 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.382459 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5b71cc-9aee-4c64-9dc8-9cfa0bf76024 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.382739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5b71cc-9aee-4c64-9dc8-9cfa0bf76024 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.383050 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5b71cc-9aee-4c64-9dc8-9cfa0bf76024 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.383670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5b71cc-9aee-4c64-9dc8-9cfa0bf76024 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.385568 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5b71cc-9aee-4c64-9dc8-9cfa0bf76024 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.388668 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b336429b-8a51-480f-96d3-74ddc6f529de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.389278 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b336429b-8a51-480f-96d3-74ddc6f529de admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.389553 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b336429b-8a51-480f-96d3-74ddc6f529de admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.389822 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b336429b-8a51-480f-96d3-74ddc6f529de admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.390477 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b336429b-8a51-480f-96d3-74ddc6f529de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.392080 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b336429b-8a51-480f-96d3-74ddc6f529de admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.405601 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 964/1944] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:29 2022] GET /identity/v3/roles => generated 1420 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:29.405601 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b336429b-8a51-480f-96d3-74ddc6f529de admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.418202 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b336429b-8a51-480f-96d3-74ddc6f529de admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.424274 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 981/1945] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:29 2022] POST /identity/v3/users => generated 280 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:29.425165 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5b71cc-9aee-4c64-9dc8-9cfa0bf76024 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.429553 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e28c8df-90f4-4a06-9413-8d3b6aa19098 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.430331 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e28c8df-90f4-4a06-9413-8d3b6aa19098 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.430674 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e28c8df-90f4-4a06-9413-8d3b6aa19098 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.431027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e28c8df-90f4-4a06-9413-8d3b6aa19098 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.431817 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e28c8df-90f4-4a06-9413-8d3b6aa19098 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.433990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e28c8df-90f4-4a06-9413-8d3b6aa19098 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.435212 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5b71cc-9aee-4c64-9dc8-9cfa0bf76024 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.442023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fc7d28-b696-413e-86a0-2be3012ed0dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.442632 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fc7d28-b696-413e-86a0-2be3012ed0dd admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.443059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fc7d28-b696-413e-86a0-2be3012ed0dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.443389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fc7d28-b696-413e-86a0-2be3012ed0dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.443969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fc7d28-b696-413e-86a0-2be3012ed0dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.445569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fc7d28-b696-413e-86a0-2be3012ed0dd admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.461537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 982/1946] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:29 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:29.461537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fc7d28-b696-413e-86a0-2be3012ed0dd admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.462135 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 965/1947] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:29 2022] POST /identity/v3/users => generated 294 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:29.463121 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e28c8df-90f4-4a06-9413-8d3b6aa19098 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.473713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e28c8df-90f4-4a06-9413-8d3b6aa19098 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.474006 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fc7d28-b696-413e-86a0-2be3012ed0dd admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.478479 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8213224f-5052-4f93-967f-b6d372eac51f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.478723 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ec1a30-5c05-46d7-bfbf-d99c04b7dbd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.479115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8213224f-5052-4f93-967f-b6d372eac51f admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.479392 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ec1a30-5c05-46d7-bfbf-d99c04b7dbd3 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.479510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8213224f-5052-4f93-967f-b6d372eac51f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.479668 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8213224f-5052-4f93-967f-b6d372eac51f admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.479758 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ec1a30-5c05-46d7-bfbf-d99c04b7dbd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.479933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ec1a30-5c05-46d7-bfbf-d99c04b7dbd3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.480291 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8213224f-5052-4f93-967f-b6d372eac51f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.480572 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ec1a30-5c05-46d7-bfbf-d99c04b7dbd3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.482119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8213224f-5052-4f93-967f-b6d372eac51f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.482260 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ec1a30-5c05-46d7-bfbf-d99c04b7dbd3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.495245 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 966/1948] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:29 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:29.497142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8213224f-5052-4f93-967f-b6d372eac51f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.502671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 983/1949] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:29 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:29.504470 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ec1a30-5c05-46d7-bfbf-d99c04b7dbd3 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.509649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8213224f-5052-4f93-967f-b6d372eac51f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.517522 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ec1a30-5c05-46d7-bfbf-d99c04b7dbd3 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.523086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35335190-041b-43f7-a55e-aed777a800de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.523778 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35335190-041b-43f7-a55e-aed777a800de admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.524084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35335190-041b-43f7-a55e-aed777a800de admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.524702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35335190-041b-43f7-a55e-aed777a800de admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.525497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35335190-041b-43f7-a55e-aed777a800de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.526289 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fea66e-45db-4f15-9801-eb782a01dd35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.527380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fea66e-45db-4f15-9801-eb782a01dd35 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.527499 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35335190-041b-43f7-a55e-aed777a800de admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.527589 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fea66e-45db-4f15-9801-eb782a01dd35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.527770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fea66e-45db-4f15-9801-eb782a01dd35 admin admin] PATH_INFO: `/v3/projects/5cbba7c8157e46e587df4846b27674e0/users/4a94affff517412e84115e913e4ab927/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.546681 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 984/1950] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:29 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:29.546681 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35335190-041b-43f7-a55e-aed777a800de admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.553765 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10fea66e-45db-4f15-9801-eb782a01dd35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5cbba7c8157e46e587df4846b27674e0, user_id=4a94affff517412e84115e913e4ab927, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.556981 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35335190-041b-43f7-a55e-aed777a800de admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.557199 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10fea66e-45db-4f15-9801-eb782a01dd35 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.564440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92281b2-9f3e-48c5-ad8e-fe9bc7b2368f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.565108 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92281b2-9f3e-48c5-ad8e-fe9bc7b2368f admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.565432 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92281b2-9f3e-48c5-ad8e-fe9bc7b2368f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.565757 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92281b2-9f3e-48c5-ad8e-fe9bc7b2368f admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.566455 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92281b2-9f3e-48c5-ad8e-fe9bc7b2368f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.568219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92281b2-9f3e-48c5-ad8e-fe9bc7b2368f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.569898 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:29.580424 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 967/1951] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:29 2022] PUT /identity/v3/projects/5cbba7c8157e46e587df4846b27674e0/users/4a94affff517412e84115e913e4ab927/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:29.581246 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fea66e-45db-4f15-9801-eb782a01dd35 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.597999 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 985/1952] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:29 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:29.599131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92281b2-9f3e-48c5-ad8e-fe9bc7b2368f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.611200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92281b2-9f3e-48c5-ad8e-fe9bc7b2368f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.690470 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd19822-e983-4505-8e57-e20feb9926b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.690681 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd19822-e983-4505-8e57-e20feb9926b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.691127 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd19822-e983-4505-8e57-e20feb9926b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.691666 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd19822-e983-4505-8e57-e20feb9926b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.692683 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd19822-e983-4505-8e57-e20feb9926b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.694694 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd19822-e983-4505-8e57-e20feb9926b6 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.698212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fea66e-45db-4f15-9801-eb782a01dd35 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.703774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8716c85c-1e59-4cbd-b7c6-e82b62616b89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.704614 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8716c85c-1e59-4cbd-b7c6-e82b62616b89 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.704889 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8716c85c-1e59-4cbd-b7c6-e82b62616b89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.705301 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8716c85c-1e59-4cbd-b7c6-e82b62616b89 admin admin] PATH_INFO: `/v3/projects/aaebdf79ec434c8587a28956b99bbc1b/users/04123e16b85b4d3796d57a607b7cd6b7/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.713275 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 986/1953] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:29 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:29.714948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf05bb8-3bc2-44d6-92a9-9d47d16e7925 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.715234 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf05bb8-3bc2-44d6-92a9-9d47d16e7925 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.718380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf05bb8-3bc2-44d6-92a9-9d47d16e7925 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.724119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8716c85c-1e59-4cbd-b7c6-e82b62616b89 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aaebdf79ec434c8587a28956b99bbc1b, user_id=04123e16b85b4d3796d57a607b7cd6b7, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.727064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8716c85c-1e59-4cbd-b7c6-e82b62616b89 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.754953 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 968/1954] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:29 2022] PUT /identity/v3/projects/aaebdf79ec434c8587a28956b99bbc1b/users/04123e16b85b4d3796d57a607b7cd6b7/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:29.756425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8716c85c-1e59-4cbd-b7c6-e82b62616b89 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.762899 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-bdf05bb8-3bc2-44d6-92a9-9d47d16e7925 None None] MFA Rules not processed for user `4a94affff517412e84115e913e4ab927`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:29.764966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-bdf05bb8-3bc2-44d6-92a9-9d47d16e7925 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:29.778993 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:29.843654 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8716c85c-1e59-4cbd-b7c6-e82b62616b89 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.851763 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae2a59-1007-4315-bcd7-3cfdec6b5640 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.853142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae2a59-1007-4315-bcd7-3cfdec6b5640 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.853496 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae2a59-1007-4315-bcd7-3cfdec6b5640 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.853779 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae2a59-1007-4315-bcd7-3cfdec6b5640 admin admin] PATH_INFO: `/v3/projects/0173fdaf0685496eb49b1583aabbb0e1/users/6b1429d80b0c459a826617783facd9d2/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.854210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 987/1955] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:29 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:29.855747 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf05bb8-3bc2-44d6-92a9-9d47d16e7925 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.881135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf05bb8-3bc2-44d6-92a9-9d47d16e7925 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:29.886669 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aae2a59-1007-4315-bcd7-3cfdec6b5640 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0173fdaf0685496eb49b1583aabbb0e1, user_id=6b1429d80b0c459a826617783facd9d2, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.887229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0489f527-3cf0-455d-b7cd-811d4f366391 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:29.888328 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0489f527-3cf0-455d-b7cd-811d4f366391 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.889173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0489f527-3cf0-455d-b7cd-811d4f366391 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.889468 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0489f527-3cf0-455d-b7cd-811d4f366391 admin admin] PATH_INFO: `/v3/projects/b56b4625387541cfa4ebb589e7404204/users/e3c0430b6f69480086dcfccbf00b1360/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.891028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aae2a59-1007-4315-bcd7-3cfdec6b5640 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.911905 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 969/1956] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:29 2022] PUT /identity/v3/projects/0173fdaf0685496eb49b1583aabbb0e1/users/6b1429d80b0c459a826617783facd9d2/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:29.913308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d6043d-797e-41f2-9673-3ab2879e4e08 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:29.913595 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d6043d-797e-41f2-9673-3ab2879e4e08 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:29.913878 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d6043d-797e-41f2-9673-3ab2879e4e08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:29.935387 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0489f527-3cf0-455d-b7cd-811d4f366391 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b56b4625387541cfa4ebb589e7404204, user_id=e3c0430b6f69480086dcfccbf00b1360, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:29.938579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0489f527-3cf0-455d-b7cd-811d4f366391 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:29.944275 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-89d6043d-797e-41f2-9673-3ab2879e4e08 None None] MFA Rules not processed for user `04123e16b85b4d3796d57a607b7cd6b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:29.945785 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-89d6043d-797e-41f2-9673-3ab2879e4e08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:29.954986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 988/1957] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:29 2022] PUT /identity/v3/projects/b56b4625387541cfa4ebb589e7404204/users/e3c0430b6f69480086dcfccbf00b1360/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:29.955715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0489f527-3cf0-455d-b7cd-811d4f366391 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:29.957228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0489f527-3cf0-455d-b7cd-811d4f366391 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:30.036245 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 970/1958] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:29 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:30.037285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd284c59-395e-427a-a239-6b015097ee8d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:30.037566 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd284c59-395e-427a-a239-6b015097ee8d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:30.037835 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd284c59-395e-427a-a239-6b015097ee8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:30.041919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0489f527-3cf0-455d-b7cd-811d4f366391 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:30.046986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446fe87d-7dcc-4671-a5af-e86e88e550d1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:30.047642 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446fe87d-7dcc-4671-a5af-e86e88e550d1 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:30.047923 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446fe87d-7dcc-4671-a5af-e86e88e550d1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:30.048195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446fe87d-7dcc-4671-a5af-e86e88e550d1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:30.058532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:30.087446 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-bd284c59-395e-427a-a239-6b015097ee8d None None] MFA Rules not processed for user `6b1429d80b0c459a826617783facd9d2`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:30.088624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-bd284c59-395e-427a-a239-6b015097ee8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:30.131673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446fe87d-7dcc-4671-a5af-e86e88e550d1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:30.133385 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446fe87d-7dcc-4671-a5af-e86e88e550d1 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:30.158396 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 989/1959] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:29 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:30.159981 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806f8612-d93c-4ded-9919-3db431741d40 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:30.162355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806f8612-d93c-4ded-9919-3db431741d40 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:30.168606 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806f8612-d93c-4ded-9919-3db431741d40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:30.180556 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 971/1960] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:30 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:30.181613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd284c59-395e-427a-a239-6b015097ee8d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:30.219503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-806f8612-d93c-4ded-9919-3db431741d40 None None] MFA Rules not processed for user `e3c0430b6f69480086dcfccbf00b1360`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:30.220896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-806f8612-d93c-4ded-9919-3db431741d40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:30.221320 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:30.305216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd284c59-395e-427a-a239-6b015097ee8d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:30.316384 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e7a335-d269-4353-9c09-abd7e1a5ea05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:30.317119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e7a335-d269-4353-9c09-abd7e1a5ea05 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:30.317497 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e7a335-d269-4353-9c09-abd7e1a5ea05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:30.317965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e7a335-d269-4353-9c09-abd7e1a5ea05 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:30.318903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e7a335-d269-4353-9c09-abd7e1a5ea05 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:30.322653 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e7a335-d269-4353-9c09-abd7e1a5ea05 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:30.335670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 990/1961] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:30 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:30.338138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806f8612-d93c-4ded-9919-3db431741d40 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:30.340181 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-806f8612-d93c-4ded-9919-3db431741d40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:30.358442 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806f8612-d93c-4ded-9919-3db431741d40 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:30.378425 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a60d4d-6f20-47b5-b903-2718b83f960c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:30.380157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a60d4d-6f20-47b5-b903-2718b83f960c service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:30.380439 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a60d4d-6f20-47b5-b903-2718b83f960c service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:30.381894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a60d4d-6f20-47b5-b903-2718b83f960c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:30.393987 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 972/1962] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:30 2022] POST /identity/v3/projects => generated 379 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:30.393987 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e7a335-d269-4353-9c09-abd7e1a5ea05 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:30.400935 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a60d4d-6f20-47b5-b903-2718b83f960c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:30.402904 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a60d4d-6f20-47b5-b903-2718b83f960c service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:30.414623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 991/1963] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:30 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:30.415484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a60d4d-6f20-47b5-b903-2718b83f960c service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:30.429217 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a60d4d-6f20-47b5-b903-2718b83f960c service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:30.504498 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e7a335-d269-4353-9c09-abd7e1a5ea05 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:30.504639 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bd2f87-073b-495c-a192-3cd35f1d53df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:30.505242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bd2f87-073b-495c-a192-3cd35f1d53df admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:30.505522 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bd2f87-073b-495c-a192-3cd35f1d53df admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:30.505791 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bd2f87-073b-495c-a192-3cd35f1d53df admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:30.506416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bd2f87-073b-495c-a192-3cd35f1d53df admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:30.508989 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bd2f87-073b-495c-a192-3cd35f1d53df admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:30.513774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d7b2f6-16d6-48d5-81b0-50b1c5c0032f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:30.514404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d7b2f6-16d6-48d5-81b0-50b1c5c0032f admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:30.514684 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d7b2f6-16d6-48d5-81b0-50b1c5c0032f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:30.514989 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d7b2f6-16d6-48d5-81b0-50b1c5c0032f admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:30.515854 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d7b2f6-16d6-48d5-81b0-50b1c5c0032f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:30.517785 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d7b2f6-16d6-48d5-81b0-50b1c5c0032f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:30.538904 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 992/1964] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:30 2022] POST /identity/v3/users => generated 296 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:30.538904 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bd2f87-073b-495c-a192-3cd35f1d53df admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:30.553136 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bd2f87-073b-495c-a192-3cd35f1d53df admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:30.558926 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48362ad1-e71d-4934-8990-ba2d63530a2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:30.559568 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48362ad1-e71d-4934-8990-ba2d63530a2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:30.559851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48362ad1-e71d-4934-8990-ba2d63530a2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:30.560123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48362ad1-e71d-4934-8990-ba2d63530a2e admin admin] PATH_INFO: `/v3/users/6b1429d80b0c459a826617783facd9d2` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:30.562587 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48362ad1-e71d-4934-8990-ba2d63530a2e admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b1429d80b0c459a826617783facd9d2)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:30.564425 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48362ad1-e71d-4934-8990-ba2d63530a2e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:30.568050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 973/1965] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:30 2022] POST /identity/v3/projects => generated 385 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:30.569609 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d7b2f6-16d6-48d5-81b0-50b1c5c0032f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:30.579009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:30.659170 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-48362ad1-e71d-4934-8990-ba2d63530a2e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b1429d80b0c459a826617783facd9d2'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:30.660052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-48362ad1-e71d-4934-8990-ba2d63530a2e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b1429d80b0c459a826617783facd9d2'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:30.665684 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d7b2f6-16d6-48d5-81b0-50b1c5c0032f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:30.672089 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be1c30d-36c1-4beb-a556-70ea726012e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:30.672947 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be1c30d-36c1-4beb-a556-70ea726012e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:30.673441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be1c30d-36c1-4beb-a556-70ea726012e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:30.673859 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be1c30d-36c1-4beb-a556-70ea726012e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:30.674796 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be1c30d-36c1-4beb-a556-70ea726012e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:30.677583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be1c30d-36c1-4beb-a556-70ea726012e3 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:30.681566 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-48362ad1-e71d-4934-8990-ba2d63530a2e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b1429d80b0c459a826617783facd9d2'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:30.683722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-48362ad1-e71d-4934-8990-ba2d63530a2e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b1429d80b0c459a826617783facd9d2'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:30.691877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 974/1966] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:30 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:30.693060 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be1c30d-36c1-4beb-a556-70ea726012e3 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:30.700727 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4be1c30d-36c1-4beb-a556-70ea726012e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:30.703806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-48362ad1-e71d-4934-8990-ba2d63530a2e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b1429d80b0c459a826617783facd9d2'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:30.721530 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be1c30d-36c1-4beb-a556-70ea726012e3 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:30.731268 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022d0985-0aa7-4409-8d44-38f38e2efad5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:30.731388 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 993/1967] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:30 2022] DELETE /identity/v3/users/6b1429d80b0c459a826617783facd9d2 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:30.731478 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022d0985-0aa7-4409-8d44-38f38e2efad5 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:30.731478 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022d0985-0aa7-4409-8d44-38f38e2efad5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:30.731478 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022d0985-0aa7-4409-8d44-38f38e2efad5 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:30.731478 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022d0985-0aa7-4409-8d44-38f38e2efad5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:30.733516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022d0985-0aa7-4409-8d44-38f38e2efad5 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:30.738769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48362ad1-e71d-4934-8990-ba2d63530a2e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:30.742457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-48362ad1-e71d-4934-8990-ba2d63530a2e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:30.774558 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 975/1968] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:30 2022] POST /identity/v3/users => generated 293 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:30.777116 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77219e43-90e0-4a05-b1f8-a6ad61cc37d9 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:30.779061 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77219e43-90e0-4a05-b1f8-a6ad61cc37d9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:30.779500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77219e43-90e0-4a05-b1f8-a6ad61cc37d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:30.806391 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:30.813097 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-77219e43-90e0-4a05-b1f8-a6ad61cc37d9 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:30.815561 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-77219e43-90e0-4a05-b1f8-a6ad61cc37d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:30.884320 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48362ad1-e71d-4934-8990-ba2d63530a2e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:30.899446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8de314-64f0-4669-8b2f-1e61a28b505b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:30.899446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8de314-64f0-4669-8b2f-1e61a28b505b admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:30.899446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8de314-64f0-4669-8b2f-1e61a28b505b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:30.899446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8de314-64f0-4669-8b2f-1e61a28b505b admin admin] PATH_INFO: `/v3/projects/a314c2edb7294a5d85d228d5cee0d5d9/users/cc5839e86b114df3b4dfe03b1a30de37/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:30.913038 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8de314-64f0-4669-8b2f-1e61a28b505b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a314c2edb7294a5d85d228d5cee0d5d9, user_id=cc5839e86b114df3b4dfe03b1a30de37, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:30.916382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8de314-64f0-4669-8b2f-1e61a28b505b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:30.933935 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 994/1969] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:30 2022] PUT /identity/v3/projects/a314c2edb7294a5d85d228d5cee0d5d9/users/cc5839e86b114df3b4dfe03b1a30de37/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:30.935564 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 976/1970] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:30.937532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8de314-64f0-4669-8b2f-1e61a28b505b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:30.947483 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77219e43-90e0-4a05-b1f8-a6ad61cc37d9 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:30.964779 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77219e43-90e0-4a05-b1f8-a6ad61cc37d9 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:31.014305 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8de314-64f0-4669-8b2f-1e61a28b505b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:31.025251 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6513c73e-ef24-41e7-925f-189c5f03a359 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:31.026104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6513c73e-ef24-41e7-925f-189c5f03a359 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:31.026550 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6513c73e-ef24-41e7-925f-189c5f03a359 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:31.027137 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6513c73e-ef24-41e7-925f-189c5f03a359 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:31.027691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5945d4-71b9-462b-abbb-04c47e7135d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:31.028136 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6513c73e-ef24-41e7-925f-189c5f03a359 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:31.028303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5945d4-71b9-462b-abbb-04c47e7135d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:31.028581 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5945d4-71b9-462b-abbb-04c47e7135d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:31.028851 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5945d4-71b9-462b-abbb-04c47e7135d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:31.030283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5945d4-71b9-462b-abbb-04c47e7135d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:31.030395 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6513c73e-ef24-41e7-925f-189c5f03a359 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:31.031779 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5945d4-71b9-462b-abbb-04c47e7135d3 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:31.042154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 977/1971] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:30 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:31.045452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5945d4-71b9-462b-abbb-04c47e7135d3 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:31.045863 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 995/1972] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:30 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:31.047052 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fe5945d4-71b9-462b-abbb-04c47e7135d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:31.047205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6513c73e-ef24-41e7-925f-189c5f03a359 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:31.058148 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6513c73e-ef24-41e7-925f-189c5f03a359 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:31.063901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f044082-73af-442a-bf43-7ba823f9104e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:31.064592 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f044082-73af-442a-bf43-7ba823f9104e admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:31.064872 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f044082-73af-442a-bf43-7ba823f9104e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:31.065147 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f044082-73af-442a-bf43-7ba823f9104e admin admin] PATH_INFO: `/v3/projects/a314c2edb7294a5d85d228d5cee0d5d9/users/cc5839e86b114df3b4dfe03b1a30de37/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:31.067318 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f044082-73af-442a-bf43-7ba823f9104e admin admin] RBAC: Authorizing `identity:create_grant(project_id=a314c2edb7294a5d85d228d5cee0d5d9, user_id=cc5839e86b114df3b4dfe03b1a30de37, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:31.070013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f044082-73af-442a-bf43-7ba823f9104e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:31.073328 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=88,keystone:DELETE=9,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:31.097481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 996/1973] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:31 2022] PUT /identity/v3/projects/a314c2edb7294a5d85d228d5cee0d5d9/users/cc5839e86b114df3b4dfe03b1a30de37/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:31.097481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f044082-73af-442a-bf43-7ba823f9104e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:31.124465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5945d4-71b9-462b-abbb-04c47e7135d3 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:31.175345 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f044082-73af-442a-bf43-7ba823f9104e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:31.180033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd7f186-2a20-4e6b-b14c-2fbe35b20090 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:31.180637 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd7f186-2a20-4e6b-b14c-2fbe35b20090 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:31.180921 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd7f186-2a20-4e6b-b14c-2fbe35b20090 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:31.181196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd7f186-2a20-4e6b-b14c-2fbe35b20090 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:31.181832 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd7f186-2a20-4e6b-b14c-2fbe35b20090 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:31.183481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd7f186-2a20-4e6b-b14c-2fbe35b20090 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:31.184992 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1ed5f8-6b7d-4c9e-b7d7-172cff1bb018 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:31.186438 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1ed5f8-6b7d-4c9e-b7d7-172cff1bb018 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:31.186713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1ed5f8-6b7d-4c9e-b7d7-172cff1bb018 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:31.187015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1ed5f8-6b7d-4c9e-b7d7-172cff1bb018 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:31.194164 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 997/1974] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:31 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:31.195409 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2ce4d0-9aea-4a75-a5fd-a616ca61d7b1 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:31.195677 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2ce4d0-9aea-4a75-a5fd-a616ca61d7b1 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:31.195982 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2ce4d0-9aea-4a75-a5fd-a616ca61d7b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:31.196971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1ed5f8-6b7d-4c9e-b7d7-172cff1bb018 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:31.198969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1ed5f8-6b7d-4c9e-b7d7-172cff1bb018 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:31.210157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 978/1975] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:31.210792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1ed5f8-6b7d-4c9e-b7d7-172cff1bb018 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:31.215221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:31.221712 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1ed5f8-6b7d-4c9e-b7d7-172cff1bb018 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:31.225203 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-af2ce4d0-9aea-4a75-a5fd-a616ca61d7b1 None None] MFA Rules not processed for user `cc5839e86b114df3b4dfe03b1a30de37`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:31.226428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-af2ce4d0-9aea-4a75-a5fd-a616ca61d7b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:31.226547 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81bf72a-471b-4484-8de0-270584b2beda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:31.227236 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81bf72a-471b-4484-8de0-270584b2beda admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:31.227519 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81bf72a-471b-4484-8de0-270584b2beda admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:31.227796 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81bf72a-471b-4484-8de0-270584b2beda admin admin] PATH_INFO: `/v3/projects/3f9277124ad14ae0be2ff58cd312dbda/users/2ea3c6571635435b83e84e45b502e442/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:31.245207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81bf72a-471b-4484-8de0-270584b2beda admin admin] RBAC: Authorizing `identity:create_grant(project_id=3f9277124ad14ae0be2ff58cd312dbda, user_id=2ea3c6571635435b83e84e45b502e442, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:31.248445 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81bf72a-471b-4484-8de0-270584b2beda admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:31.266323 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 979/1976] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:31 2022] PUT /identity/v3/projects/3f9277124ad14ae0be2ff58cd312dbda/users/2ea3c6571635435b83e84e45b502e442/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:31.279554 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e3fc57-ef70-4815-ad3a-92ec5e524a3c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:31.279747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e3fc57-ef70-4815-ad3a-92ec5e524a3c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:31.280496 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e3fc57-ef70-4815-ad3a-92ec5e524a3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:31.305679 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 998/1977] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:31 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:31.318799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e2e3fc57-ef70-4815-ad3a-92ec5e524a3c None None] MFA Rules not processed for user `2ea3c6571635435b83e84e45b502e442`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:31.320337 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e2e3fc57-ef70-4815-ad3a-92ec5e524a3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:31.320988 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2ce4d0-9aea-4a75-a5fd-a616ca61d7b1 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:31.322942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-af2ce4d0-9aea-4a75-a5fd-a616ca61d7b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:31.389053 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2ce4d0-9aea-4a75-a5fd-a616ca61d7b1 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:31.415601 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 980/1978] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:31 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:19:31.422493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e3fc57-ef70-4815-ad3a-92ec5e524a3c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:31.433747 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:31.458050 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39361de1-398f-4d5d-b0aa-83d68c75160f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:31.459233 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39361de1-398f-4d5d-b0aa-83d68c75160f service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:31.459517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39361de1-398f-4d5d-b0aa-83d68c75160f service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:31.460012 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39361de1-398f-4d5d-b0aa-83d68c75160f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:31.540674 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e3fc57-ef70-4815-ad3a-92ec5e524a3c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:31.546837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2316bb8a-2363-44bf-9613-9c2066cb79a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:31.547498 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2316bb8a-2363-44bf-9613-9c2066cb79a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:31.547776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2316bb8a-2363-44bf-9613-9c2066cb79a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:31.548046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2316bb8a-2363-44bf-9613-9c2066cb79a6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:31.548682 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2316bb8a-2363-44bf-9613-9c2066cb79a6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:31.550582 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2316bb8a-2363-44bf-9613-9c2066cb79a6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:31.581240 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39361de1-398f-4d5d-b0aa-83d68c75160f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:31.582350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 981/1979] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:31 2022] POST /identity/v3/projects => generated 385 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:31.582675 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39361de1-398f-4d5d-b0aa-83d68c75160f service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:31.586200 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2316bb8a-2363-44bf-9613-9c2066cb79a6 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:31.593235 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 999/1980] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:31 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 273 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:19:31.619270 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39361de1-398f-4d5d-b0aa-83d68c75160f service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:31.622308 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-39361de1-398f-4d5d-b0aa-83d68c75160f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:31.660823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39361de1-398f-4d5d-b0aa-83d68c75160f service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:31.708666 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:31.735987 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2316bb8a-2363-44bf-9613-9c2066cb79a6 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:31.742170 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f393d29d-4765-4d5b-9663-0c663849e010 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:31.742802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f393d29d-4765-4d5b-9663-0c663849e010 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:31.743208 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f393d29d-4765-4d5b-9663-0c663849e010 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:31.743485 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f393d29d-4765-4d5b-9663-0c663849e010 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:31.744128 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f393d29d-4765-4d5b-9663-0c663849e010 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:31.747666 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f393d29d-4765-4d5b-9663-0c663849e010 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:31.771193 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab33f8a5-6613-40c7-ae82-40b9e2786fb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:31.771919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab33f8a5-6613-40c7-ae82-40b9e2786fb8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:31.772516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab33f8a5-6613-40c7-ae82-40b9e2786fb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:31.772800 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab33f8a5-6613-40c7-ae82-40b9e2786fb8 admin admin] PATH_INFO: `/v3/projects/0173fdaf0685496eb49b1583aabbb0e1` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:31.774210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab33f8a5-6613-40c7-ae82-40b9e2786fb8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0173fdaf0685496eb49b1583aabbb0e1)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:31.776276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab33f8a5-6613-40c7-ae82-40b9e2786fb8 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:31.785528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 982/1981] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:31 2022] POST /identity/v3/users => generated 299 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:31.789875 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f393d29d-4765-4d5b-9663-0c663849e010 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:31.804537 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f393d29d-4765-4d5b-9663-0c663849e010 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:31.809415 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bded46be-24e3-487f-b782-9a78aea7613a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:31.810064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bded46be-24e3-487f-b782-9a78aea7613a admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:31.811915 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bded46be-24e3-487f-b782-9a78aea7613a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:31.812373 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bded46be-24e3-487f-b782-9a78aea7613a admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:31.813072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bded46be-24e3-487f-b782-9a78aea7613a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:31.815011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bded46be-24e3-487f-b782-9a78aea7613a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:31.834663 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 983/1982] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:31 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:31.836338 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bded46be-24e3-487f-b782-9a78aea7613a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:31.839605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ab33f8a5-6613-40c7-ae82-40b9e2786fb8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0173fdaf0685496eb49b1583aabbb0e1', 'request_id': 'req-ab33f8a5-6613-40c7-ae82-40b9e2786fb8'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:31.847655 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ab33f8a5-6613-40c7-ae82-40b9e2786fb8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0173fdaf0685496eb49b1583aabbb0e1', 'request_id': 'req-ab33f8a5-6613-40c7-ae82-40b9e2786fb8'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:31.850475 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bded46be-24e3-487f-b782-9a78aea7613a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:31.857330 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ab33f8a5-6613-40c7-ae82-40b9e2786fb8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0173fdaf0685496eb49b1583aabbb0e1', 'request_id': 'req-ab33f8a5-6613-40c7-ae82-40b9e2786fb8'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:31.874594 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ab33f8a5-6613-40c7-ae82-40b9e2786fb8 admin admin] The token cache is being invalidate because project 0173fdaf0685496eb49b1583aabbb0e1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:31.875577 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ab33f8a5-6613-40c7-ae82-40b9e2786fb8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:31.876387 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ab33f8a5-6613-40c7-ae82-40b9e2786fb8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:31.878573 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1000/1983] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:31 2022] DELETE /identity/v3/projects/0173fdaf0685496eb49b1583aabbb0e1 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:31.887792 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f9752d-f8d3-4c3f-9949-3f3d853d3e80 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:31.888151 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f9752d-f8d3-4c3f-9949-3f3d853d3e80 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:31.888594 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f9752d-f8d3-4c3f-9949-3f3d853d3e80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:31.916979 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b5f9752d-f8d3-4c3f-9949-3f3d853d3e80 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:31.918172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b5f9752d-f8d3-4c3f-9949-3f3d853d3e80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:31.932057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:31.944235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab79917-81ff-4f3b-b02f-72081b59781e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:31.945480 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab79917-81ff-4f3b-b02f-72081b59781e admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:31.945826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab79917-81ff-4f3b-b02f-72081b59781e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:31.946191 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab79917-81ff-4f3b-b02f-72081b59781e admin admin] PATH_INFO: `/v3/projects/e7dd7a401a434c81b1befe34ee1b3804/users/f40301242b1043f18563380ea9bc6eaf/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:31.965276 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab79917-81ff-4f3b-b02f-72081b59781e admin admin] RBAC: Authorizing `identity:create_grant(project_id=e7dd7a401a434c81b1befe34ee1b3804, user_id=f40301242b1043f18563380ea9bc6eaf, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:31.974345 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab79917-81ff-4f3b-b02f-72081b59781e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:31.994171 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 984/1984] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:31 2022] PUT /identity/v3/projects/e7dd7a401a434c81b1befe34ee1b3804/users/f40301242b1043f18563380ea9bc6eaf/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:31.995073 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab79917-81ff-4f3b-b02f-72081b59781e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:31.996782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9ab79917-81ff-4f3b-b02f-72081b59781e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:32.048864 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1001/1985] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:32.054782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0320ad35-155f-45d8-aae3-d0d9db42890a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:32.055290 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0320ad35-155f-45d8-aae3-d0d9db42890a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:32.055726 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0320ad35-155f-45d8-aae3-d0d9db42890a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:32.105498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-0320ad35-155f-45d8-aae3-d0d9db42890a None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:32.112837 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0320ad35-155f-45d8-aae3-d0d9db42890a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:32.113418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab79917-81ff-4f3b-b02f-72081b59781e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:32.122233 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931475ff-8299-4dcd-867e-1c88c6e642ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:32.123110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931475ff-8299-4dcd-867e-1c88c6e642ad admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:32.123639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931475ff-8299-4dcd-867e-1c88c6e642ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:32.123921 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931475ff-8299-4dcd-867e-1c88c6e642ad admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:32.124794 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931475ff-8299-4dcd-867e-1c88c6e642ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:32.126888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931475ff-8299-4dcd-867e-1c88c6e642ad admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:32.142434 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 985/1986] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:31 2022] GET /identity/v3/roles => generated 1420 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:19:32.143549 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931475ff-8299-4dcd-867e-1c88c6e642ad admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:32.143960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1002/1987] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:32.149479 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-931475ff-8299-4dcd-867e-1c88c6e642ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:32.160763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0320ad35-155f-45d8-aae3-d0d9db42890a None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:32.210457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0320ad35-155f-45d8-aae3-d0d9db42890a None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:32.237269 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb33411-4510-4ce9-8e88-b7d6f97ad36a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:32.239625 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb33411-4510-4ce9-8e88-b7d6f97ad36a admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:32.240034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb33411-4510-4ce9-8e88-b7d6f97ad36a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:32.240901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb33411-4510-4ce9-8e88-b7d6f97ad36a admin admin] PATH_INFO: `/v3/projects/e7dd7a401a434c81b1befe34ee1b3804/users/f40301242b1043f18563380ea9bc6eaf/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:32.246040 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb33411-4510-4ce9-8e88-b7d6f97ad36a admin admin] RBAC: Authorizing `identity:create_grant(project_id=e7dd7a401a434c81b1befe34ee1b3804, user_id=f40301242b1043f18563380ea9bc6eaf, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:32.252335 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb33411-4510-4ce9-8e88-b7d6f97ad36a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:32.284534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:32.292310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1003/1988] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:32 2022] PUT /identity/v3/projects/e7dd7a401a434c81b1befe34ee1b3804/users/f40301242b1043f18563380ea9bc6eaf/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:32.294201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb33411-4510-4ce9-8e88-b7d6f97ad36a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:32.325963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931475ff-8299-4dcd-867e-1c88c6e642ad admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:32.396931 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a33431e-3fde-429c-becc-d59234dac819 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc5839e86b114df3b4dfe03b1a30de37', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a314c2edb7294a5d85d228d5cee0d5d9', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:32.397852 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a33431e-3fde-429c-becc-d59234dac819 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:32.398278 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a33431e-3fde-429c-becc-d59234dac819 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:32.398694 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a33431e-3fde-429c-becc-d59234dac819 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:32.399503 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a33431e-3fde-429c-becc-d59234dac819 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:19:32.399951 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a33431e-3fde-429c-becc-d59234dac819 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:32.402410 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a33431e-3fde-429c-becc-d59234dac819 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:32.418544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 986/1989] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:19:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:19:32.423709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915d5e63-4bfd-451f-9ec0-add975e15beb None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:32.424000 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915d5e63-4bfd-451f-9ec0-add975e15beb None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:32.424274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915d5e63-4bfd-451f-9ec0-add975e15beb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:32.435734 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb33411-4510-4ce9-8e88-b7d6f97ad36a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:32.442322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339b4fa8-4db8-4d55-99fe-e643fd3b2595 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:32.443012 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339b4fa8-4db8-4d55-99fe-e643fd3b2595 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:32.443294 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339b4fa8-4db8-4d55-99fe-e643fd3b2595 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:32.443567 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339b4fa8-4db8-4d55-99fe-e643fd3b2595 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:32.444209 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339b4fa8-4db8-4d55-99fe-e643fd3b2595 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:19:32.461382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339b4fa8-4db8-4d55-99fe-e643fd3b2595 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:32.461382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339b4fa8-4db8-4d55-99fe-e643fd3b2595 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:32.462588 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1004/1990] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:19:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:19:32.463274 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339b4fa8-4db8-4d55-99fe-e643fd3b2595 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:32.473467 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-915d5e63-4bfd-451f-9ec0-add975e15beb None None] MFA Rules not processed for user `f40301242b1043f18563380ea9bc6eaf`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:32.474735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339b4fa8-4db8-4d55-99fe-e643fd3b2595 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:32.474877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-915d5e63-4bfd-451f-9ec0-add975e15beb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:32.479909 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286c9a1b-279c-464b-b477-94956b9405b4 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc5839e86b114df3b4dfe03b1a30de37', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a314c2edb7294a5d85d228d5cee0d5d9', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:32.480526 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286c9a1b-279c-464b-b477-94956b9405b4 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:32.480806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286c9a1b-279c-464b-b477-94956b9405b4 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:32.481077 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286c9a1b-279c-464b-b477-94956b9405b4 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:32.481711 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286c9a1b-279c-464b-b477-94956b9405b4 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:32.483676 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286c9a1b-279c-464b-b477-94956b9405b4 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:32.516333 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1005/1991] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:32 2022] POST /identity/v3/projects => generated 355 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:32.517944 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286c9a1b-279c-464b-b477-94956b9405b4 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:32.554131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 987/1992] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:32 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:32.560014 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915d5e63-4bfd-451f-9ec0-add975e15beb None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:32.577125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915d5e63-4bfd-451f-9ec0-add975e15beb None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:32.617375 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286c9a1b-279c-464b-b477-94956b9405b4 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:32.623839 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e28b788-4ac0-4ef4-9151-84ff15a06ea7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:32.624688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e28b788-4ac0-4ef4-9151-84ff15a06ea7 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:32.624968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e28b788-4ac0-4ef4-9151-84ff15a06ea7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:32.625236 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e28b788-4ac0-4ef4-9151-84ff15a06ea7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:32.625871 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e28b788-4ac0-4ef4-9151-84ff15a06ea7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:32.627799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e28b788-4ac0-4ef4-9151-84ff15a06ea7 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:32.647318 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:32.671148 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1006/1993] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:32 2022] POST /identity/v3/projects => generated 373 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:32.671148 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e28b788-4ac0-4ef4-9151-84ff15a06ea7 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:32.673370 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9e28b788-4ac0-4ef4-9151-84ff15a06ea7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:32.676155 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e06f235-6bb7-448c-ab0f-8b950a85863f tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc5839e86b114df3b4dfe03b1a30de37', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a314c2edb7294a5d85d228d5cee0d5d9', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:32.676793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e06f235-6bb7-448c-ab0f-8b950a85863f tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:32.677064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e06f235-6bb7-448c-ab0f-8b950a85863f tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:32.677486 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e06f235-6bb7-448c-ab0f-8b950a85863f tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:32.677987 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e06f235-6bb7-448c-ab0f-8b950a85863f tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:19:32.678291 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e06f235-6bb7-448c-ab0f-8b950a85863f tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:32.680069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e06f235-6bb7-448c-ab0f-8b950a85863f tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:32.699012 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 988/1994] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:19:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:19:32.701711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e06f235-6bb7-448c-ab0f-8b950a85863f tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:32.713343 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e06f235-6bb7-448c-ab0f-8b950a85863f tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:32.724193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:32.775926 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e28b788-4ac0-4ef4-9151-84ff15a06ea7 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:32.780013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b05f2c-1dda-4fe2-8e60-0c3ee54a941c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:32.780621 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b05f2c-1dda-4fe2-8e60-0c3ee54a941c service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:32.780897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b05f2c-1dda-4fe2-8e60-0c3ee54a941c service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:32.781167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b05f2c-1dda-4fe2-8e60-0c3ee54a941c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:32.801399 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa57a487-0d09-4c7f-9ad5-7ef87c72b07b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:32.802015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa57a487-0d09-4c7f-9ad5-7ef87c72b07b admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:32.802590 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa57a487-0d09-4c7f-9ad5-7ef87c72b07b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:32.802897 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa57a487-0d09-4c7f-9ad5-7ef87c72b07b admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:32.803536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa57a487-0d09-4c7f-9ad5-7ef87c72b07b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:32.805468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa57a487-0d09-4c7f-9ad5-7ef87c72b07b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:32.828186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 989/1995] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:32 2022] POST /identity/v3/users => generated 287 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:32.829151 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa57a487-0d09-4c7f-9ad5-7ef87c72b07b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:32.843058 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b05f2c-1dda-4fe2-8e60-0c3ee54a941c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:32.844486 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b05f2c-1dda-4fe2-8e60-0c3ee54a941c service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:32.853882 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1007/1996] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:32 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:32.854661 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b05f2c-1dda-4fe2-8e60-0c3ee54a941c service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:32.865235 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b05f2c-1dda-4fe2-8e60-0c3ee54a941c service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:32.869925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5279c550-516d-402b-8ca0-a113d3bfb6d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:32.870518 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5279c550-516d-402b-8ca0-a113d3bfb6d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:32.870798 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5279c550-516d-402b-8ca0-a113d3bfb6d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:32.871108 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5279c550-516d-402b-8ca0-a113d3bfb6d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:32.871745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5279c550-516d-402b-8ca0-a113d3bfb6d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:32.873335 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5279c550-516d-402b-8ca0-a113d3bfb6d0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:32.884687 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1008/1997] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:32 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:32.889278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5279c550-516d-402b-8ca0-a113d3bfb6d0 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:32.901241 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa57a487-0d09-4c7f-9ad5-7ef87c72b07b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:32.904091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5279c550-516d-402b-8ca0-a113d3bfb6d0 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:32.905974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cf7f04-90bb-48fe-b614-516d07cb0100 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc5839e86b114df3b4dfe03b1a30de37', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'a314c2edb7294a5d85d228d5cee0d5d9', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:32.906609 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cf7f04-90bb-48fe-b614-516d07cb0100 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:32.906918 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cf7f04-90bb-48fe-b614-516d07cb0100 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:32.907224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cf7f04-90bb-48fe-b614-516d07cb0100 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] PATH_INFO: `/v3/projects/125e4c122e3946f9ac9c64c203c514c1` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:32.908413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cf7f04-90bb-48fe-b614-516d07cb0100 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] RBAC: Authorizing `identity:delete_project(project_id=125e4c122e3946f9ac9c64c203c514c1)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:32.910550 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cf7f04-90bb-48fe-b614-516d07cb0100 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:32.911252 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc642e11-a51e-48e2-848e-02caacd59a82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:32.912039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc642e11-a51e-48e2-848e-02caacd59a82 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:32.912388 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc642e11-a51e-48e2-848e-02caacd59a82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:32.912808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc642e11-a51e-48e2-848e-02caacd59a82 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:32.913425 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc642e11-a51e-48e2-848e-02caacd59a82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:32.915232 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc642e11-a51e-48e2-848e-02caacd59a82 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:32.929770 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1009/1998] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:32 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:19:32.930607 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc642e11-a51e-48e2-848e-02caacd59a82 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:32.941431 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc642e11-a51e-48e2-848e-02caacd59a82 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:32.947222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14008d0d-abb6-4345-9a9c-7887021b5ee4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:32.947913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14008d0d-abb6-4345-9a9c-7887021b5ee4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:32.948190 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14008d0d-abb6-4345-9a9c-7887021b5ee4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:32.948461 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14008d0d-abb6-4345-9a9c-7887021b5ee4 admin admin] PATH_INFO: `/v3/projects/b0422bd9612349b8a0cbcae6f1a588b7/users/93409ebeeab6464abf34f26ebd37a2a9/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:32.964637 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14008d0d-abb6-4345-9a9c-7887021b5ee4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b0422bd9612349b8a0cbcae6f1a588b7, user_id=93409ebeeab6464abf34f26ebd37a2a9, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:32.967020 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e2cf7f04-90bb-48fe-b614-516d07cb0100 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '125e4c122e3946f9ac9c64c203c514c1', 'request_id': 'req-e2cf7f04-90bb-48fe-b614-516d07cb0100'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:32.967352 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14008d0d-abb6-4345-9a9c-7887021b5ee4 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:32.973154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e2cf7f04-90bb-48fe-b614-516d07cb0100 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '125e4c122e3946f9ac9c64c203c514c1', 'request_id': 'req-e2cf7f04-90bb-48fe-b614-516d07cb0100'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:32.980308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e2cf7f04-90bb-48fe-b614-516d07cb0100 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '125e4c122e3946f9ac9c64c203c514c1', 'request_id': 'req-e2cf7f04-90bb-48fe-b614-516d07cb0100'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:32.984162 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1010/1999] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:32 2022] PUT /identity/v3/projects/b0422bd9612349b8a0cbcae6f1a588b7/users/93409ebeeab6464abf34f26ebd37a2a9/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:32.994316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e2cf7f04-90bb-48fe-b614-516d07cb0100 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] The token cache is being invalidate because project 125e4c122e3946f9ac9c64c203c514c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:32.994699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e2cf7f04-90bb-48fe-b614-516d07cb0100 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:32.995123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c57cbd0-8861-428e-b7e4-f2a4883c7905 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:32.995461 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c57cbd0-8861-428e-b7e4-f2a4883c7905 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:32.995739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c57cbd0-8861-428e-b7e4-f2a4883c7905 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:32.996045 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e2cf7f04-90bb-48fe-b614-516d07cb0100 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:32.997855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 990/2000] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:32 2022] DELETE /identity/v3/projects/125e4c122e3946f9ac9c64c203c514c1 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:33.028113 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9c57cbd0-8861-428e-b7e4-f2a4883c7905 None None] MFA Rules not processed for user `93409ebeeab6464abf34f26ebd37a2a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:33.029410 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9c57cbd0-8861-428e-b7e4-f2a4883c7905 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:33.078032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:33.112213 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1011/2001] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:32 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:19:33.141251 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cf7f04-90bb-48fe-b614-516d07cb0100 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:33.142160 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e2cf7f04-90bb-48fe-b614-516d07cb0100 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:33.176624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:33.204390 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cf7f04-90bb-48fe-b614-516d07cb0100 tempest-VolumeQuotaClassesTest-725614436 tempest-VolumeQuotaClassesTest-725614436-project-admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:33.254148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59730053-81da-4e33-ae01-71e62ee1b29d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:33.254903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59730053-81da-4e33-ae01-71e62ee1b29d service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:33.255215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59730053-81da-4e33-ae01-71e62ee1b29d service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:33.255484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59730053-81da-4e33-ae01-71e62ee1b29d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:33.264964 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59730053-81da-4e33-ae01-71e62ee1b29d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:33.266724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59730053-81da-4e33-ae01-71e62ee1b29d service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:33.278282 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 991/2002] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:33 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:33.551780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c57cbd0-8861-428e-b7e4-f2a4883c7905 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:33.553783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9c57cbd0-8861-428e-b7e4-f2a4883c7905 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:33.670126 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c57cbd0-8861-428e-b7e4-f2a4883c7905 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:33.677649 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b00e47-240b-4f38-b097-ade96862b0d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:33.678878 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b00e47-240b-4f38-b097-ade96862b0d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:33.679231 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b00e47-240b-4f38-b097-ade96862b0d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:33.679585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b00e47-240b-4f38-b097-ade96862b0d6 admin admin] PATH_INFO: `/v3/users/04123e16b85b4d3796d57a607b7cd6b7` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:33.681023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b00e47-240b-4f38-b097-ade96862b0d6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=04123e16b85b4d3796d57a607b7cd6b7)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:33.683263 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b00e47-240b-4f38-b097-ade96862b0d6 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:33.738581 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-15b00e47-240b-4f38-b097-ade96862b0d6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '04123e16b85b4d3796d57a607b7cd6b7'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:33.739445 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-15b00e47-240b-4f38-b097-ade96862b0d6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '04123e16b85b4d3796d57a607b7cd6b7'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:33.747272 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-15b00e47-240b-4f38-b097-ade96862b0d6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '04123e16b85b4d3796d57a607b7cd6b7'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:33.748462 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-15b00e47-240b-4f38-b097-ade96862b0d6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '04123e16b85b4d3796d57a607b7cd6b7'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:33.760889 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-15b00e47-240b-4f38-b097-ade96862b0d6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '04123e16b85b4d3796d57a607b7cd6b7'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:33.774837 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1012/2003] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:33 2022] DELETE /identity/v3/users/04123e16b85b4d3796d57a607b7cd6b7 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:33.779632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0d149a-81c6-4100-bd3b-323c8747b71d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:33.780124 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0d149a-81c6-4100-bd3b-323c8747b71d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:33.780306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0d149a-81c6-4100-bd3b-323c8747b71d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:33.808625 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6a0d149a-81c6-4100-bd3b-323c8747b71d None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:33.811331 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6a0d149a-81c6-4100-bd3b-323c8747b71d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:33.884290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 992/2004] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:33.899814 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b00e47-240b-4f38-b097-ade96862b0d6 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:33.901317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-15b00e47-240b-4f38-b097-ade96862b0d6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:33.997466 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b00e47-240b-4f38-b097-ade96862b0d6 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:34.002061 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e55c6b-5f23-4dcc-975f-0b1b69a0635b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:34.002679 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e55c6b-5f23-4dcc-975f-0b1b69a0635b service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:34.003008 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e55c6b-5f23-4dcc-975f-0b1b69a0635b service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:34.003334 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e55c6b-5f23-4dcc-975f-0b1b69a0635b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:34.016515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e55c6b-5f23-4dcc-975f-0b1b69a0635b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:34.017584 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e55c6b-5f23-4dcc-975f-0b1b69a0635b service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:34.028694 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1013/2005] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:34.098463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0d149a-81c6-4100-bd3b-323c8747b71d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:34.099989 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6a0d149a-81c6-4100-bd3b-323c8747b71d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:34.159785 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0d149a-81c6-4100-bd3b-323c8747b71d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:34.163443 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c1244b-16e3-4569-85bc-c8769abd63f2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:34.164060 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c1244b-16e3-4569-85bc-c8769abd63f2 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:34.164340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c1244b-16e3-4569-85bc-c8769abd63f2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:34.164615 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c1244b-16e3-4569-85bc-c8769abd63f2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:34.166343 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-25c1244b-16e3-4569-85bc-c8769abd63f2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:34.217592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:34.233342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c1244b-16e3-4569-85bc-c8769abd63f2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:34.234808 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c1244b-16e3-4569-85bc-c8769abd63f2 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:34.243676 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 993/2006] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:34 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:19:34.304975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e55c6b-5f23-4dcc-975f-0b1b69a0635b service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:34.306394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-91e55c6b-5f23-4dcc-975f-0b1b69a0635b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:34.322601 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e55c6b-5f23-4dcc-975f-0b1b69a0635b service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:34.327659 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0599f9-5da6-48b3-9af1-a0fc7aba5b43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:34.328291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0599f9-5da6-48b3-9af1-a0fc7aba5b43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:34.328568 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0599f9-5da6-48b3-9af1-a0fc7aba5b43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:34.328839 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0599f9-5da6-48b3-9af1-a0fc7aba5b43 admin admin] PATH_INFO: `/v3/projects/aaebdf79ec434c8587a28956b99bbc1b` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:34.329950 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0599f9-5da6-48b3-9af1-a0fc7aba5b43 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aaebdf79ec434c8587a28956b99bbc1b)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:34.331874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0599f9-5da6-48b3-9af1-a0fc7aba5b43 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:34.333504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:34.392615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ee0599f9-5da6-48b3-9af1-a0fc7aba5b43 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aaebdf79ec434c8587a28956b99bbc1b', 'request_id': 'req-ee0599f9-5da6-48b3-9af1-a0fc7aba5b43'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:34.400269 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ee0599f9-5da6-48b3-9af1-a0fc7aba5b43 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aaebdf79ec434c8587a28956b99bbc1b', 'request_id': 'req-ee0599f9-5da6-48b3-9af1-a0fc7aba5b43'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:34.407909 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ee0599f9-5da6-48b3-9af1-a0fc7aba5b43 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aaebdf79ec434c8587a28956b99bbc1b', 'request_id': 'req-ee0599f9-5da6-48b3-9af1-a0fc7aba5b43'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:34.425752 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ee0599f9-5da6-48b3-9af1-a0fc7aba5b43 admin admin] The token cache is being invalidate because project aaebdf79ec434c8587a28956b99bbc1b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:34.426043 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ee0599f9-5da6-48b3-9af1-a0fc7aba5b43 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:34.426662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ee0599f9-5da6-48b3-9af1-a0fc7aba5b43 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:34.428192 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1014/2007] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:34 2022] DELETE /identity/v3/projects/aaebdf79ec434c8587a28956b99bbc1b => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:34.432394 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c1244b-16e3-4569-85bc-c8769abd63f2 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:34.444120 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c1244b-16e3-4569-85bc-c8769abd63f2 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:34.523252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d91292-2082-42a2-99c4-ae67c61458d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:34.523908 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d91292-2082-42a2-99c4-ae67c61458d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:34.524288 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d91292-2082-42a2-99c4-ae67c61458d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:34.524497 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d91292-2082-42a2-99c4-ae67c61458d6 admin admin] PATH_INFO: `/v3/users/cc5839e86b114df3b4dfe03b1a30de37` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:34.525624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d91292-2082-42a2-99c4-ae67c61458d6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cc5839e86b114df3b4dfe03b1a30de37)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:34.527654 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d91292-2082-42a2-99c4-ae67c61458d6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:34.606484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c0d91292-2082-42a2-99c4-ae67c61458d6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cc5839e86b114df3b4dfe03b1a30de37'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:34.607269 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c0d91292-2082-42a2-99c4-ae67c61458d6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cc5839e86b114df3b4dfe03b1a30de37'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:34.615455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c0d91292-2082-42a2-99c4-ae67c61458d6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cc5839e86b114df3b4dfe03b1a30de37'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:34.616351 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c0d91292-2082-42a2-99c4-ae67c61458d6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cc5839e86b114df3b4dfe03b1a30de37'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:34.623534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c0d91292-2082-42a2-99c4-ae67c61458d6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cc5839e86b114df3b4dfe03b1a30de37'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:34.641375 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 994/2008] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:34 2022] DELETE /identity/v3/users/cc5839e86b114df3b4dfe03b1a30de37 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:19:34.921908 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0599f9-5da6-48b3-9af1-a0fc7aba5b43 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:34.923433 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ee0599f9-5da6-48b3-9af1-a0fc7aba5b43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:34.937723 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0599f9-5da6-48b3-9af1-a0fc7aba5b43 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:34.993634 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef0a18e-19f2-4c6e-bae8-b8dd2233d8be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:34.994195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef0a18e-19f2-4c6e-bae8-b8dd2233d8be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:34.994473 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef0a18e-19f2-4c6e-bae8-b8dd2233d8be admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:34.994743 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef0a18e-19f2-4c6e-bae8-b8dd2233d8be admin admin] PATH_INFO: `/v3/projects/a314c2edb7294a5d85d228d5cee0d5d9` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:34.995882 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef0a18e-19f2-4c6e-bae8-b8dd2233d8be admin admin] RBAC: Authorizing `identity:delete_project(project_id=a314c2edb7294a5d85d228d5cee0d5d9)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:34.998285 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef0a18e-19f2-4c6e-bae8-b8dd2233d8be admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:35.062286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9ef0a18e-19f2-4c6e-bae8-b8dd2233d8be admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a314c2edb7294a5d85d228d5cee0d5d9', 'request_id': 'req-9ef0a18e-19f2-4c6e-bae8-b8dd2233d8be'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:35.076415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9ef0a18e-19f2-4c6e-bae8-b8dd2233d8be admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a314c2edb7294a5d85d228d5cee0d5d9', 'request_id': 'req-9ef0a18e-19f2-4c6e-bae8-b8dd2233d8be'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:35.084860 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9ef0a18e-19f2-4c6e-bae8-b8dd2233d8be admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a314c2edb7294a5d85d228d5cee0d5d9', 'request_id': 'req-9ef0a18e-19f2-4c6e-bae8-b8dd2233d8be'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:35.101523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9ef0a18e-19f2-4c6e-bae8-b8dd2233d8be admin admin] The token cache is being invalidate because project a314c2edb7294a5d85d228d5cee0d5d9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:35.102051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9ef0a18e-19f2-4c6e-bae8-b8dd2233d8be admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:35.102718 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9ef0a18e-19f2-4c6e-bae8-b8dd2233d8be admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:35.104698 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1015/2009] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:34 2022] DELETE /identity/v3/projects/a314c2edb7294a5d85d228d5cee0d5d9 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:35.109836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b226b1-4dfd-4e86-976c-351b51bc8126 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:35.110394 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b226b1-4dfd-4e86-976c-351b51bc8126 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:35.110767 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b226b1-4dfd-4e86-976c-351b51bc8126 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:35.135433 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-84b226b1-4dfd-4e86-976c-351b51bc8126 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:35.136886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-84b226b1-4dfd-4e86-976c-351b51bc8126 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:35.216086 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 995/2010] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:35.223687 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7b83ac-64f4-4302-9957-21e74fa5ec22 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:35.224632 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7b83ac-64f4-4302-9957-21e74fa5ec22 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:35.224987 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7b83ac-64f4-4302-9957-21e74fa5ec22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:35.250908 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-0f7b83ac-64f4-4302-9957-21e74fa5ec22 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:35.252210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0f7b83ac-64f4-4302-9957-21e74fa5ec22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:35.262316 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1016/2011] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:35.267972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b226b1-4dfd-4e86-976c-351b51bc8126 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:35.278761 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:35.288026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b226b1-4dfd-4e86-976c-351b51bc8126 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:35.302635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4c37b5-04f1-4bfe-8d8b-99e844f7d1a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:35.303414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4c37b5-04f1-4bfe-8d8b-99e844f7d1a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:35.304378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4c37b5-04f1-4bfe-8d8b-99e844f7d1a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:35.304378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4c37b5-04f1-4bfe-8d8b-99e844f7d1a8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:35.304848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4c37b5-04f1-4bfe-8d8b-99e844f7d1a8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:19:35.305385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4c37b5-04f1-4bfe-8d8b-99e844f7d1a8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:35.307147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4c37b5-04f1-4bfe-8d8b-99e844f7d1a8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:35.323237 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 996/2012] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:19:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:19:35.325859 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7b83ac-64f4-4302-9957-21e74fa5ec22 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:35.335026 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7b83ac-64f4-4302-9957-21e74fa5ec22 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:35.340660 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a306678a-fc95-42eb-b6c8-a892c6b53e59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:35.340817 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a306678a-fc95-42eb-b6c8-a892c6b53e59 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:35.341072 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a306678a-fc95-42eb-b6c8-a892c6b53e59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:35.341366 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a306678a-fc95-42eb-b6c8-a892c6b53e59 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:35.342005 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a306678a-fc95-42eb-b6c8-a892c6b53e59 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:35.343926 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a306678a-fc95-42eb-b6c8-a892c6b53e59 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:35.356590 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:35.375518 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1017/2013] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:35 2022] POST /identity/v3/projects => generated 375 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:35.380021 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4c37b5-04f1-4bfe-8d8b-99e844f7d1a8 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:35.391527 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4c37b5-04f1-4bfe-8d8b-99e844f7d1a8 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:35.471444 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857d444f-dfbc-4724-a4a4-3ce018942b21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:35.472310 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d444f-dfbc-4724-a4a4-3ce018942b21 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:35.472770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d444f-dfbc-4724-a4a4-3ce018942b21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:35.473309 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d444f-dfbc-4724-a4a4-3ce018942b21 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:35.474022 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857d444f-dfbc-4724-a4a4-3ce018942b21 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:35.477642 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857d444f-dfbc-4724-a4a4-3ce018942b21 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:35.504018 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 997/2014] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:35 2022] POST /identity/v3/users => generated 288 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:35.512054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a306678a-fc95-42eb-b6c8-a892c6b53e59 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:35.525288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a306678a-fc95-42eb-b6c8-a892c6b53e59 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:35.530493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1d56dc-3e38-4440-9c60-459087fd19cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:35.531135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1d56dc-3e38-4440-9c60-459087fd19cd admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:35.531560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1d56dc-3e38-4440-9c60-459087fd19cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:35.531836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1d56dc-3e38-4440-9c60-459087fd19cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:35.532480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1d56dc-3e38-4440-9c60-459087fd19cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:35.534057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1d56dc-3e38-4440-9c60-459087fd19cd admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:35.550460 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1018/2015] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:35 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:35.554465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857d444f-dfbc-4724-a4a4-3ce018942b21 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:35.566537 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857d444f-dfbc-4724-a4a4-3ce018942b21 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:35.571758 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ddadd1-4dd0-47ba-8a00-953cb7c16a09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:35.572367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ddadd1-4dd0-47ba-8a00-953cb7c16a09 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:35.572650 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ddadd1-4dd0-47ba-8a00-953cb7c16a09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:35.572939 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ddadd1-4dd0-47ba-8a00-953cb7c16a09 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:35.573576 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ddadd1-4dd0-47ba-8a00-953cb7c16a09 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:35.575253 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ddadd1-4dd0-47ba-8a00-953cb7c16a09 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:35.589499 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 998/2016] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:35 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:35.594477 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1d56dc-3e38-4440-9c60-459087fd19cd admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:35.604967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1d56dc-3e38-4440-9c60-459087fd19cd admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:35.610127 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eabc362-24dc-4031-97fb-48cc39b97d75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:35.610870 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eabc362-24dc-4031-97fb-48cc39b97d75 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:35.611222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eabc362-24dc-4031-97fb-48cc39b97d75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:35.611562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eabc362-24dc-4031-97fb-48cc39b97d75 admin admin] PATH_INFO: `/v3/projects/af238b24545740078b837d22fa6e3779/users/4e5f9e9d4ff24d7ca437761007cfea2b/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:35.634094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eabc362-24dc-4031-97fb-48cc39b97d75 admin admin] RBAC: Authorizing `identity:create_grant(project_id=af238b24545740078b837d22fa6e3779, user_id=4e5f9e9d4ff24d7ca437761007cfea2b, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:35.637695 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eabc362-24dc-4031-97fb-48cc39b97d75 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:35.654481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1019/2017] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:35 2022] PUT /identity/v3/projects/af238b24545740078b837d22fa6e3779/users/4e5f9e9d4ff24d7ca437761007cfea2b/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:35.666204 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2ff91a-c070-4850-9f5a-bbdca49d7a03 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:35.666560 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2ff91a-c070-4850-9f5a-bbdca49d7a03 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:35.666947 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2ff91a-c070-4850-9f5a-bbdca49d7a03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:35.704476 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-fd2ff91a-c070-4850-9f5a-bbdca49d7a03 None None] MFA Rules not processed for user `4e5f9e9d4ff24d7ca437761007cfea2b`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:35.707139 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fd2ff91a-c070-4850-9f5a-bbdca49d7a03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:35.773065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84487581-6203-4d15-ab96-0b19d3c6db31 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:35.773359 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84487581-6203-4d15-ab96-0b19d3c6db31 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:35.773642 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84487581-6203-4d15-ab96-0b19d3c6db31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:35.796269 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-84487581-6203-4d15-ab96-0b19d3c6db31 None None] MFA Rules not processed for user `93409ebeeab6464abf34f26ebd37a2a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:35.799203 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-84487581-6203-4d15-ab96-0b19d3c6db31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:35.807950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 999/2018] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:35 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:19:35.813855 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2ff91a-c070-4850-9f5a-bbdca49d7a03 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:35.882596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1020/2019] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:35 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:19:35.895791 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2ff91a-c070-4850-9f5a-bbdca49d7a03 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:35.905206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93dfd52-9430-489d-ad84-d652d6b1b6bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:35.905206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93dfd52-9430-489d-ad84-d652d6b1b6bc admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:35.905738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93dfd52-9430-489d-ad84-d652d6b1b6bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:35.906190 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93dfd52-9430-489d-ad84-d652d6b1b6bc admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:35.907286 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93dfd52-9430-489d-ad84-d652d6b1b6bc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:35.910039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93dfd52-9430-489d-ad84-d652d6b1b6bc admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:35.916827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:35.944887 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1000/2020] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:35 2022] POST /identity/v3/projects => generated 375 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:35.949239 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84487581-6203-4d15-ab96-0b19d3c6db31 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:35.961396 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84487581-6203-4d15-ab96-0b19d3c6db31 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:36.024008 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f72a87c-a0cd-4258-bdd7-853f9d3eaa72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:36.024864 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f72a87c-a0cd-4258-bdd7-853f9d3eaa72 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:36.025144 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f72a87c-a0cd-4258-bdd7-853f9d3eaa72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:36.025433 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f72a87c-a0cd-4258-bdd7-853f9d3eaa72 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:36.026076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f72a87c-a0cd-4258-bdd7-853f9d3eaa72 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:36.028531 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f72a87c-a0cd-4258-bdd7-853f9d3eaa72 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:36.055764 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1021/2021] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:35 2022] POST /identity/v3/users => generated 294 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:36.057427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93dfd52-9430-489d-ad84-d652d6b1b6bc admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:36.069248 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93dfd52-9430-489d-ad84-d652d6b1b6bc admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:36.074402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba60d833-ebf3-4d07-af4b-ae7835f029fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:36.075451 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba60d833-ebf3-4d07-af4b-ae7835f029fa admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:36.075562 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba60d833-ebf3-4d07-af4b-ae7835f029fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:36.076500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba60d833-ebf3-4d07-af4b-ae7835f029fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:36.077323 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba60d833-ebf3-4d07-af4b-ae7835f029fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:36.079048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba60d833-ebf3-4d07-af4b-ae7835f029fa admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:36.090420 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1001/2022] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:36 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:36.094699 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f72a87c-a0cd-4258-bdd7-853f9d3eaa72 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:36.105639 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f72a87c-a0cd-4258-bdd7-853f9d3eaa72 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:36.111242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa3874c-bfab-42a2-bf98-9ff6fb972bdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:36.111959 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa3874c-bfab-42a2-bf98-9ff6fb972bdc admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:36.112249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa3874c-bfab-42a2-bf98-9ff6fb972bdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:36.112528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa3874c-bfab-42a2-bf98-9ff6fb972bdc admin admin] PATH_INFO: `/v3/projects/d068a4a79c964cbfb187e05aff78f675/users/f7033cb6b52440158fb6c5e703e7ab9a/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:36.130701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa3874c-bfab-42a2-bf98-9ff6fb972bdc admin admin] RBAC: Authorizing `identity:create_grant(project_id=d068a4a79c964cbfb187e05aff78f675, user_id=f7033cb6b52440158fb6c5e703e7ab9a, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:36.133807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa3874c-bfab-42a2-bf98-9ff6fb972bdc admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:36.149847 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1022/2023] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:36 2022] PUT /identity/v3/projects/d068a4a79c964cbfb187e05aff78f675/users/f7033cb6b52440158fb6c5e703e7ab9a/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:19:36.153990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba60d833-ebf3-4d07-af4b-ae7835f029fa admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:36.164419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba60d833-ebf3-4d07-af4b-ae7835f029fa admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:36.225484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee31972a-1b19-4eb7-8dbe-5a13760e6ec3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:36.226142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee31972a-1b19-4eb7-8dbe-5a13760e6ec3 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:36.226427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee31972a-1b19-4eb7-8dbe-5a13760e6ec3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:36.226702 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee31972a-1b19-4eb7-8dbe-5a13760e6ec3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:36.227382 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee31972a-1b19-4eb7-8dbe-5a13760e6ec3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:36.229003 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee31972a-1b19-4eb7-8dbe-5a13760e6ec3 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:36.240357 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1002/2024] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:36 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:36.244537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa3874c-bfab-42a2-bf98-9ff6fb972bdc admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:36.255045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa3874c-bfab-42a2-bf98-9ff6fb972bdc admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:36.259682 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfde2ea8-335e-45fe-a583-5c33b64c0b65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:36.260352 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfde2ea8-335e-45fe-a583-5c33b64c0b65 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:36.260631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfde2ea8-335e-45fe-a583-5c33b64c0b65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:36.260914 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfde2ea8-335e-45fe-a583-5c33b64c0b65 admin admin] PATH_INFO: `/v3/projects/d068a4a79c964cbfb187e05aff78f675/users/f7033cb6b52440158fb6c5e703e7ab9a/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:36.262831 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfde2ea8-335e-45fe-a583-5c33b64c0b65 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d068a4a79c964cbfb187e05aff78f675, user_id=f7033cb6b52440158fb6c5e703e7ab9a, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:36.265562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfde2ea8-335e-45fe-a583-5c33b64c0b65 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:36.276476 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1023/2025] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:36 2022] PUT /identity/v3/projects/d068a4a79c964cbfb187e05aff78f675/users/f7033cb6b52440158fb6c5e703e7ab9a/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:36.287143 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c847559-bb8b-476c-a4a2-5b516dbe0ac3 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:36.287435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c847559-bb8b-476c-a4a2-5b516dbe0ac3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:36.287716 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c847559-bb8b-476c-a4a2-5b516dbe0ac3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:36.316751 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-1c847559-bb8b-476c-a4a2-5b516dbe0ac3 None None] MFA Rules not processed for user `f7033cb6b52440158fb6c5e703e7ab9a`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:36.317986 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1c847559-bb8b-476c-a4a2-5b516dbe0ac3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:36.388919 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1003/2026] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:36 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:36.399611 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfde2ea8-335e-45fe-a583-5c33b64c0b65 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:36.401513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-dfde2ea8-335e-45fe-a583-5c33b64c0b65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:36.423311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:36.469910 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfde2ea8-335e-45fe-a583-5c33b64c0b65 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:36.473513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1948b3-b0ec-4fe6-8a08-8aece7fd6ac9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:36.474123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1948b3-b0ec-4fe6-8a08-8aece7fd6ac9 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:36.474399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1948b3-b0ec-4fe6-8a08-8aece7fd6ac9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:36.474670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1948b3-b0ec-4fe6-8a08-8aece7fd6ac9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:36.540412 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1948b3-b0ec-4fe6-8a08-8aece7fd6ac9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:36.542040 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de1948b3-b0ec-4fe6-8a08-8aece7fd6ac9 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:36.555029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1024/2027] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:36 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:38.205061 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c847559-bb8b-476c-a4a2-5b516dbe0ac3 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:38.207137 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1c847559-bb8b-476c-a4a2-5b516dbe0ac3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:38.374368 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:38.417475 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c847559-bb8b-476c-a4a2-5b516dbe0ac3 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:38.429440 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e13cedd-6a13-4fb2-acd1-8e28752415f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:38.429805 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e13cedd-6a13-4fb2-acd1-8e28752415f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:38.430092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e13cedd-6a13-4fb2-acd1-8e28752415f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:38.430367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e13cedd-6a13-4fb2-acd1-8e28752415f1 admin admin] PATH_INFO: `/v3/users/4a94affff517412e84115e913e4ab927` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:38.435622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e13cedd-6a13-4fb2-acd1-8e28752415f1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4a94affff517412e84115e913e4ab927)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:38.444306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e13cedd-6a13-4fb2-acd1-8e28752415f1 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:38.563460 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0e13cedd-6a13-4fb2-acd1-8e28752415f1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4a94affff517412e84115e913e4ab927'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:38.564176 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0e13cedd-6a13-4fb2-acd1-8e28752415f1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4a94affff517412e84115e913e4ab927'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:38.577028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0e13cedd-6a13-4fb2-acd1-8e28752415f1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4a94affff517412e84115e913e4ab927'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:38.579923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0e13cedd-6a13-4fb2-acd1-8e28752415f1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4a94affff517412e84115e913e4ab927'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:38.592354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0e13cedd-6a13-4fb2-acd1-8e28752415f1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4a94affff517412e84115e913e4ab927'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:38.619852 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1004/2028] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:38 2022] DELETE /identity/v3/users/4a94affff517412e84115e913e4ab927 => generated 0 bytes in 412 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:38.622157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41533f4e-b083-4a2e-acd8-b6fc615badb8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:38.623308 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41533f4e-b083-4a2e-acd8-b6fc615badb8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:38.623446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41533f4e-b083-4a2e-acd8-b6fc615badb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:38.645742 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-41533f4e-b083-4a2e-acd8-b6fc615badb8 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:38.646988 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-41533f4e-b083-4a2e-acd8-b6fc615badb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:38.723829 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1025/2029] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:38.738383 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e13cedd-6a13-4fb2-acd1-8e28752415f1 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:38.739871 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0e13cedd-6a13-4fb2-acd1-8e28752415f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:38.749891 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41533f4e-b083-4a2e-acd8-b6fc615badb8 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:38.751938 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-41533f4e-b083-4a2e-acd8-b6fc615badb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:38.824919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41533f4e-b083-4a2e-acd8-b6fc615badb8 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:38.831950 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0b2075-3535-4fbe-a1a1-8d2e6dc70b7f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:38.832667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0b2075-3535-4fbe-a1a1-8d2e6dc70b7f service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:38.832985 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0b2075-3535-4fbe-a1a1-8d2e6dc70b7f service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:38.833273 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0b2075-3535-4fbe-a1a1-8d2e6dc70b7f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:38.837303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2a0b2075-3535-4fbe-a1a1-8d2e6dc70b7f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:38.853749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e13cedd-6a13-4fb2-acd1-8e28752415f1 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:38.857685 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e586d23-ea5f-46d9-a4db-3c99d5a3ff86 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:38.858673 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e586d23-ea5f-46d9-a4db-3c99d5a3ff86 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:38.858673 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e586d23-ea5f-46d9-a4db-3c99d5a3ff86 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:38.858894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e586d23-ea5f-46d9-a4db-3c99d5a3ff86 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:38.872205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e586d23-ea5f-46d9-a4db-3c99d5a3ff86 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:38.874223 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e586d23-ea5f-46d9-a4db-3c99d5a3ff86 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:38.883057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:38.889436 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1005/2030] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:38.939051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0b2075-3535-4fbe-a1a1-8d2e6dc70b7f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:38.940731 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0b2075-3535-4fbe-a1a1-8d2e6dc70b7f service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:38.956386 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1026/2031] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:38 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:39.464186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e586d23-ea5f-46d9-a4db-3c99d5a3ff86 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:39.465668 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3e586d23-ea5f-46d9-a4db-3c99d5a3ff86 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:39.488357 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e586d23-ea5f-46d9-a4db-3c99d5a3ff86 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:39.503473 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1d018a-dc0c-4206-9f4a-c6c6f28ed789 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:39.504126 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1d018a-dc0c-4206-9f4a-c6c6f28ed789 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:39.504404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1d018a-dc0c-4206-9f4a-c6c6f28ed789 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:39.504677 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1d018a-dc0c-4206-9f4a-c6c6f28ed789 admin admin] PATH_INFO: `/v3/users/4e5f9e9d4ff24d7ca437761007cfea2b` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:39.505837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1d018a-dc0c-4206-9f4a-c6c6f28ed789 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4e5f9e9d4ff24d7ca437761007cfea2b)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:39.507827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1d018a-dc0c-4206-9f4a-c6c6f28ed789 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:39.605547 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3a1d018a-dc0c-4206-9f4a-c6c6f28ed789 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4e5f9e9d4ff24d7ca437761007cfea2b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:39.606466 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3a1d018a-dc0c-4206-9f4a-c6c6f28ed789 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4e5f9e9d4ff24d7ca437761007cfea2b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:39.622448 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3a1d018a-dc0c-4206-9f4a-c6c6f28ed789 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4e5f9e9d4ff24d7ca437761007cfea2b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:39.624157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3a1d018a-dc0c-4206-9f4a-c6c6f28ed789 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4e5f9e9d4ff24d7ca437761007cfea2b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:39.636835 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3a1d018a-dc0c-4206-9f4a-c6c6f28ed789 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4e5f9e9d4ff24d7ca437761007cfea2b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:39.647491 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:39.706105 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1006/2032] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:39 2022] DELETE /identity/v3/users/4e5f9e9d4ff24d7ca437761007cfea2b => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:39.710608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442dd2fc-0cfc-4ae0-968e-2cd117b72381 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:39.711131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442dd2fc-0cfc-4ae0-968e-2cd117b72381 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:39.711213 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442dd2fc-0cfc-4ae0-968e-2cd117b72381 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:39.760551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-442dd2fc-0cfc-4ae0-968e-2cd117b72381 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:39.762102 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-442dd2fc-0cfc-4ae0-968e-2cd117b72381 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:39.847140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1027/2033] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:39.873103 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1d018a-dc0c-4206-9f4a-c6c6f28ed789 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:39.874546 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3a1d018a-dc0c-4206-9f4a-c6c6f28ed789 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:39.975414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1d018a-dc0c-4206-9f4a-c6c6f28ed789 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:39.986556 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78449246-33d0-44be-adb1-5fcc770c94b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:39.987642 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78449246-33d0-44be-adb1-5fcc770c94b0 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:39.988760 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78449246-33d0-44be-adb1-5fcc770c94b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:39.988760 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78449246-33d0-44be-adb1-5fcc770c94b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:40.020176 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78449246-33d0-44be-adb1-5fcc770c94b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:40.020176 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78449246-33d0-44be-adb1-5fcc770c94b0 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:40.030952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1007/2034] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:40.035611 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442dd2fc-0cfc-4ae0-968e-2cd117b72381 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:40.037099 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-442dd2fc-0cfc-4ae0-968e-2cd117b72381 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:40.053211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442dd2fc-0cfc-4ae0-968e-2cd117b72381 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:40.058542 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24b25a5-d90a-4b23-9fc9-766bc76b2496 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:40.059273 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24b25a5-d90a-4b23-9fc9-766bc76b2496 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:40.059602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24b25a5-d90a-4b23-9fc9-766bc76b2496 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:40.060490 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24b25a5-d90a-4b23-9fc9-766bc76b2496 admin admin] PATH_INFO: `/v3/projects/5cbba7c8157e46e587df4846b27674e0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:40.061914 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24b25a5-d90a-4b23-9fc9-766bc76b2496 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5cbba7c8157e46e587df4846b27674e0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:40.065145 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24b25a5-d90a-4b23-9fc9-766bc76b2496 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:40.153208 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e24b25a5-d90a-4b23-9fc9-766bc76b2496 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5cbba7c8157e46e587df4846b27674e0', 'request_id': 'req-e24b25a5-d90a-4b23-9fc9-766bc76b2496'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:40.163877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e24b25a5-d90a-4b23-9fc9-766bc76b2496 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5cbba7c8157e46e587df4846b27674e0', 'request_id': 'req-e24b25a5-d90a-4b23-9fc9-766bc76b2496'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:40.173590 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e24b25a5-d90a-4b23-9fc9-766bc76b2496 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5cbba7c8157e46e587df4846b27674e0', 'request_id': 'req-e24b25a5-d90a-4b23-9fc9-766bc76b2496'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:40.194617 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e24b25a5-d90a-4b23-9fc9-766bc76b2496 admin admin] The token cache is being invalidate because project 5cbba7c8157e46e587df4846b27674e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:40.194811 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e24b25a5-d90a-4b23-9fc9-766bc76b2496 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:40.195493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e24b25a5-d90a-4b23-9fc9-766bc76b2496 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:40.197176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1028/2035] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:40 2022] DELETE /identity/v3/projects/5cbba7c8157e46e587df4846b27674e0 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:40.202364 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c87e9b-cfb0-49cc-a9a2-66f2eaac1dfb None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:40.202647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c87e9b-cfb0-49cc-a9a2-66f2eaac1dfb None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:40.202950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c87e9b-cfb0-49cc-a9a2-66f2eaac1dfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:40.224844 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c0c87e9b-cfb0-49cc-a9a2-66f2eaac1dfb None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:40.226064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c0c87e9b-cfb0-49cc-a9a2-66f2eaac1dfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:40.312694 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1008/2036] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:40.314635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b44ea2-9113-48dd-8058-fb25bd105ad8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:40.315009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b44ea2-9113-48dd-8058-fb25bd105ad8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:40.315287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b44ea2-9113-48dd-8058-fb25bd105ad8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:40.339789 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-71b44ea2-9113-48dd-8058-fb25bd105ad8 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:40.342027 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-71b44ea2-9113-48dd-8058-fb25bd105ad8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:40.355325 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1029/2037] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:40.357234 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c87e9b-cfb0-49cc-a9a2-66f2eaac1dfb None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:40.371403 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c87e9b-cfb0-49cc-a9a2-66f2eaac1dfb None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:40.376148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66826a8-8f74-451e-9c16-66169fc211be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:40.376824 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66826a8-8f74-451e-9c16-66169fc211be admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:40.377167 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66826a8-8f74-451e-9c16-66169fc211be admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:40.377507 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66826a8-8f74-451e-9c16-66169fc211be admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:40.378219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66826a8-8f74-451e-9c16-66169fc211be admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:19:40.378594 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66826a8-8f74-451e-9c16-66169fc211be admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:40.380283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66826a8-8f74-451e-9c16-66169fc211be admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:40.397660 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1009/2038] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:19:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:19:40.400227 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b44ea2-9113-48dd-8058-fb25bd105ad8 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:40.423290 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b44ea2-9113-48dd-8058-fb25bd105ad8 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:40.445805 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa441c1a-bd5e-4068-b415-b678bd2979b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:40.446396 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa441c1a-bd5e-4068-b415-b678bd2979b8 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:40.447287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa441c1a-bd5e-4068-b415-b678bd2979b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:40.447568 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa441c1a-bd5e-4068-b415-b678bd2979b8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:40.448385 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa441c1a-bd5e-4068-b415-b678bd2979b8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:40.450091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa441c1a-bd5e-4068-b415-b678bd2979b8 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:40.509920 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1030/2039] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:40 2022] POST /identity/v3/projects => generated 373 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:40.513714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66826a8-8f74-451e-9c16-66169fc211be admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:40.540082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66826a8-8f74-451e-9c16-66169fc211be admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:40.689103 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed75a7e-43f6-46f2-a6e1-35ced5668585 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:40.690123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed75a7e-43f6-46f2-a6e1-35ced5668585 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:40.690699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed75a7e-43f6-46f2-a6e1-35ced5668585 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:40.691312 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed75a7e-43f6-46f2-a6e1-35ced5668585 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:40.692168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed75a7e-43f6-46f2-a6e1-35ced5668585 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:40.695154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed75a7e-43f6-46f2-a6e1-35ced5668585 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:40.739164 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1010/2040] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:40 2022] POST /identity/v3/users => generated 287 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:40.743213 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa441c1a-bd5e-4068-b415-b678bd2979b8 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:40.758791 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa441c1a-bd5e-4068-b415-b678bd2979b8 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:40.776151 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc9d25a-ac47-42a3-94d9-12a3695b24d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:40.777672 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc9d25a-ac47-42a3-94d9-12a3695b24d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:40.778242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc9d25a-ac47-42a3-94d9-12a3695b24d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:40.779423 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc9d25a-ac47-42a3-94d9-12a3695b24d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:40.780817 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc9d25a-ac47-42a3-94d9-12a3695b24d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:40.783241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc9d25a-ac47-42a3-94d9-12a3695b24d3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:40.801064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed75a7e-43f6-46f2-a6e1-35ced5668585 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:40.803370 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2ed75a7e-43f6-46f2-a6e1-35ced5668585 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:40.819967 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed75a7e-43f6-46f2-a6e1-35ced5668585 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:40.820288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1031/2041] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:40 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:19:40.821836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc9d25a-ac47-42a3-94d9-12a3695b24d3 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:40.827047 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddc8271-8349-40f4-ba7e-f5d678aa1ad4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:40.828031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddc8271-8349-40f4-ba7e-f5d678aa1ad4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:40.828997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddc8271-8349-40f4-ba7e-f5d678aa1ad4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:40.829372 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddc8271-8349-40f4-ba7e-f5d678aa1ad4 admin admin] PATH_INFO: `/v3/projects/af238b24545740078b837d22fa6e3779` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:40.830763 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddc8271-8349-40f4-ba7e-f5d678aa1ad4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=af238b24545740078b837d22fa6e3779)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:40.832856 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddc8271-8349-40f4-ba7e-f5d678aa1ad4 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:40.836033 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:40.836329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc9d25a-ac47-42a3-94d9-12a3695b24d3 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:40.844167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2673616e-4587-4df8-b362-6e030170852c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:40.845087 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2673616e-4587-4df8-b362-6e030170852c admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:40.845631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2673616e-4587-4df8-b362-6e030170852c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:40.846218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2673616e-4587-4df8-b362-6e030170852c admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:40.847124 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2673616e-4587-4df8-b362-6e030170852c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:40.848963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2673616e-4587-4df8-b362-6e030170852c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:40.850786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:40.876762 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1032/2042] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:40 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:40.884316 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2673616e-4587-4df8-b362-6e030170852c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:40.898827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2673616e-4587-4df8-b362-6e030170852c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:40.919104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cddc8271-8349-40f4-ba7e-f5d678aa1ad4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'af238b24545740078b837d22fa6e3779', 'request_id': 'req-cddc8271-8349-40f4-ba7e-f5d678aa1ad4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:40.941153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cddc8271-8349-40f4-ba7e-f5d678aa1ad4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'af238b24545740078b837d22fa6e3779', 'request_id': 'req-cddc8271-8349-40f4-ba7e-f5d678aa1ad4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:40.951980 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cddc8271-8349-40f4-ba7e-f5d678aa1ad4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'af238b24545740078b837d22fa6e3779', 'request_id': 'req-cddc8271-8349-40f4-ba7e-f5d678aa1ad4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:40.982205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cddc8271-8349-40f4-ba7e-f5d678aa1ad4 admin admin] The token cache is being invalidate because project af238b24545740078b837d22fa6e3779 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:40.983288 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cddc8271-8349-40f4-ba7e-f5d678aa1ad4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:40.983980 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cddc8271-8349-40f4-ba7e-f5d678aa1ad4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:40.990402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1011/2043] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:40 2022] DELETE /identity/v3/projects/af238b24545740078b837d22fa6e3779 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:40.991164 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c44de7-ed95-48f8-a3ed-aac52b60fbf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:40.992074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c44de7-ed95-48f8-a3ed-aac52b60fbf5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:40.992595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c44de7-ed95-48f8-a3ed-aac52b60fbf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:40.993503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c44de7-ed95-48f8-a3ed-aac52b60fbf5 admin admin] PATH_INFO: `/v3/projects/de31bd722d4f419d9269b519af7f793a/users/85178dc2280c4a54970fc207d1746608/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:40.993988 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddc8271-8349-40f4-ba7e-f5d678aa1ad4 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:40.995945 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cddc8271-8349-40f4-ba7e-f5d678aa1ad4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:41.021268 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c44de7-ed95-48f8-a3ed-aac52b60fbf5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=de31bd722d4f419d9269b519af7f793a, user_id=85178dc2280c4a54970fc207d1746608, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:41.022698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddc8271-8349-40f4-ba7e-f5d678aa1ad4 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:41.025133 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c44de7-ed95-48f8-a3ed-aac52b60fbf5 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:41.033347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c98377a-fdaa-4838-b3a3-33b0b7132ed9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:41.034463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c98377a-fdaa-4838-b3a3-33b0b7132ed9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:41.034984 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c98377a-fdaa-4838-b3a3-33b0b7132ed9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:41.035144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c98377a-fdaa-4838-b3a3-33b0b7132ed9 admin admin] PATH_INFO: `/v3/users/f7033cb6b52440158fb6c5e703e7ab9a` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:41.036835 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c98377a-fdaa-4838-b3a3-33b0b7132ed9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f7033cb6b52440158fb6c5e703e7ab9a)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:41.039593 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c98377a-fdaa-4838-b3a3-33b0b7132ed9 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:41.048597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1033/2044] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:40 2022] PUT /identity/v3/projects/de31bd722d4f419d9269b519af7f793a/users/85178dc2280c4a54970fc207d1746608/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:41.054302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a01c66e-8e39-4e21-be61-e63afb2dbd8a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:41.054831 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a01c66e-8e39-4e21-be61-e63afb2dbd8a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:41.055425 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a01c66e-8e39-4e21-be61-e63afb2dbd8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:41.085013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4a01c66e-8e39-4e21-be61-e63afb2dbd8a None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:41.086583 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4a01c66e-8e39-4e21-be61-e63afb2dbd8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:41.158910 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9c98377a-fdaa-4838-b3a3-33b0b7132ed9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f7033cb6b52440158fb6c5e703e7ab9a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:41.163597 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9c98377a-fdaa-4838-b3a3-33b0b7132ed9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f7033cb6b52440158fb6c5e703e7ab9a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:41.182407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9c98377a-fdaa-4838-b3a3-33b0b7132ed9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f7033cb6b52440158fb6c5e703e7ab9a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:41.183239 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9c98377a-fdaa-4838-b3a3-33b0b7132ed9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f7033cb6b52440158fb6c5e703e7ab9a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:41.221517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9c98377a-fdaa-4838-b3a3-33b0b7132ed9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f7033cb6b52440158fb6c5e703e7ab9a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:41.249854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1034/2045] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:19:41.259356 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1012/2046] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:40 2022] DELETE /identity/v3/users/f7033cb6b52440158fb6c5e703e7ab9a => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:41.298015 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a01c66e-8e39-4e21-be61-e63afb2dbd8a None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:41.299896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4a01c66e-8e39-4e21-be61-e63afb2dbd8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:41.394781 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a01c66e-8e39-4e21-be61-e63afb2dbd8a None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:41.467456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:41.536279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca53ec07-b787-4732-9fe5-6f35cb11188f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:41.540674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca53ec07-b787-4732-9fe5-6f35cb11188f service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:41.541072 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca53ec07-b787-4732-9fe5-6f35cb11188f service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:41.543771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca53ec07-b787-4732-9fe5-6f35cb11188f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:41.720964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca53ec07-b787-4732-9fe5-6f35cb11188f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:41.722565 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca53ec07-b787-4732-9fe5-6f35cb11188f service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:41.745596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1035/2047] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 448 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:19:41.940465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c98377a-fdaa-4838-b3a3-33b0b7132ed9 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:41.941978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9c98377a-fdaa-4838-b3a3-33b0b7132ed9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:41.968198 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c98377a-fdaa-4838-b3a3-33b0b7132ed9 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:41.980049 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576db7c1-8cdf-4ea2-bb58-fa56c0322f8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:41.981616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576db7c1-8cdf-4ea2-bb58-fa56c0322f8c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:41.982164 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576db7c1-8cdf-4ea2-bb58-fa56c0322f8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:41.982626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576db7c1-8cdf-4ea2-bb58-fa56c0322f8c admin admin] PATH_INFO: `/v3/users/e3c0430b6f69480086dcfccbf00b1360` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:41.984555 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576db7c1-8cdf-4ea2-bb58-fa56c0322f8c admin admin] RBAC: Authorizing `identity:delete_user(user_id=e3c0430b6f69480086dcfccbf00b1360)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:41.987737 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576db7c1-8cdf-4ea2-bb58-fa56c0322f8c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:42.069050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-576db7c1-8cdf-4ea2-bb58-fa56c0322f8c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e3c0430b6f69480086dcfccbf00b1360'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:42.070663 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-576db7c1-8cdf-4ea2-bb58-fa56c0322f8c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e3c0430b6f69480086dcfccbf00b1360'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:42.083419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-576db7c1-8cdf-4ea2-bb58-fa56c0322f8c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e3c0430b6f69480086dcfccbf00b1360'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:42.090955 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-576db7c1-8cdf-4ea2-bb58-fa56c0322f8c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e3c0430b6f69480086dcfccbf00b1360'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:42.106431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=72,keystone:DELETE=25,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:42.119349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-576db7c1-8cdf-4ea2-bb58-fa56c0322f8c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e3c0430b6f69480086dcfccbf00b1360'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:42.146502 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1013/2048] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:41 2022] DELETE /identity/v3/users/e3c0430b6f69480086dcfccbf00b1360 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:42.151296 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e814f9d9-b4d8-4fc6-b72a-387f2d993a8a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:42.151651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e814f9d9-b4d8-4fc6-b72a-387f2d993a8a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:42.151857 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e814f9d9-b4d8-4fc6-b72a-387f2d993a8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:42.190645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e814f9d9-b4d8-4fc6-b72a-387f2d993a8a None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:42.192046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e814f9d9-b4d8-4fc6-b72a-387f2d993a8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:42.304849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1036/2049] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:19:42.357997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576db7c1-8cdf-4ea2-bb58-fa56c0322f8c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:42.361624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-576db7c1-8cdf-4ea2-bb58-fa56c0322f8c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:42.474341 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576db7c1-8cdf-4ea2-bb58-fa56c0322f8c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:42.478455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2f0494-a6bb-494a-9910-e0ee086d5f86 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:42.479272 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2f0494-a6bb-494a-9910-e0ee086d5f86 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:42.479627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2f0494-a6bb-494a-9910-e0ee086d5f86 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:42.480000 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2f0494-a6bb-494a-9910-e0ee086d5f86 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:42.498546 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2f0494-a6bb-494a-9910-e0ee086d5f86 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:42.500259 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2f0494-a6bb-494a-9910-e0ee086d5f86 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:42.516881 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1014/2050] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:43.178639 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e814f9d9-b4d8-4fc6-b72a-387f2d993a8a None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:43.180127 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e814f9d9-b4d8-4fc6-b72a-387f2d993a8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:43.206875 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e814f9d9-b4d8-4fc6-b72a-387f2d993a8a None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:43.217105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e686cdc-2555-4f10-8486-0c3744c902d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:43.217477 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e686cdc-2555-4f10-8486-0c3744c902d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:43.217718 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e686cdc-2555-4f10-8486-0c3744c902d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:43.217991 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e686cdc-2555-4f10-8486-0c3744c902d3 admin admin] PATH_INFO: `/v3/projects/d068a4a79c964cbfb187e05aff78f675` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:43.221269 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e686cdc-2555-4f10-8486-0c3744c902d3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d068a4a79c964cbfb187e05aff78f675)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:43.225228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e686cdc-2555-4f10-8486-0c3744c902d3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:43.370723 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3e686cdc-2555-4f10-8486-0c3744c902d3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd068a4a79c964cbfb187e05aff78f675', 'request_id': 'req-3e686cdc-2555-4f10-8486-0c3744c902d3'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:43.395900 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3e686cdc-2555-4f10-8486-0c3744c902d3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd068a4a79c964cbfb187e05aff78f675', 'request_id': 'req-3e686cdc-2555-4f10-8486-0c3744c902d3'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:43.424013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3e686cdc-2555-4f10-8486-0c3744c902d3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd068a4a79c964cbfb187e05aff78f675', 'request_id': 'req-3e686cdc-2555-4f10-8486-0c3744c902d3'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:43.453699 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3e686cdc-2555-4f10-8486-0c3744c902d3 admin admin] The token cache is being invalidate because project d068a4a79c964cbfb187e05aff78f675 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:43.453927 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3e686cdc-2555-4f10-8486-0c3744c902d3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:43.455214 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3e686cdc-2555-4f10-8486-0c3744c902d3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:43.460509 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1037/2051] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:43 2022] DELETE /identity/v3/projects/d068a4a79c964cbfb187e05aff78f675 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:43.464971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d33f6f-3297-4467-bd11-da629b809703 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:43.466691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d33f6f-3297-4467-bd11-da629b809703 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:43.467887 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d33f6f-3297-4467-bd11-da629b809703 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:43.499738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-62d33f6f-3297-4467-bd11-da629b809703 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:43.501164 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-62d33f6f-3297-4467-bd11-da629b809703 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:43.613476 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1015/2052] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:43.615251 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c06796d-32cc-4fac-8fcd-9423e2418afc None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:43.615544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c06796d-32cc-4fac-8fcd-9423e2418afc None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:43.615814 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c06796d-32cc-4fac-8fcd-9423e2418afc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:43.671663 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-6c06796d-32cc-4fac-8fcd-9423e2418afc None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:43.673060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6c06796d-32cc-4fac-8fcd-9423e2418afc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:43.690161 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1038/2053] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:43.693890 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d33f6f-3297-4467-bd11-da629b809703 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:43.704730 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d33f6f-3297-4467-bd11-da629b809703 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:43.711795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c44cc3a-6370-43fa-85e9-bf282b7edd7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:43.713280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c44cc3a-6370-43fa-85e9-bf282b7edd7c admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:43.713445 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c44cc3a-6370-43fa-85e9-bf282b7edd7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:43.713765 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c44cc3a-6370-43fa-85e9-bf282b7edd7c admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:43.715005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c44cc3a-6370-43fa-85e9-bf282b7edd7c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:19:43.715365 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c44cc3a-6370-43fa-85e9-bf282b7edd7c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:43.717799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c44cc3a-6370-43fa-85e9-bf282b7edd7c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:43.765358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1016/2054] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:19:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:19:43.767752 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c06796d-32cc-4fac-8fcd-9423e2418afc None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:43.773397 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:43.786167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c06796d-32cc-4fac-8fcd-9423e2418afc None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:43.803247 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc456c70-2493-46cb-8467-7bd5dd369c78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:43.804384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc456c70-2493-46cb-8467-7bd5dd369c78 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:43.804958 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc456c70-2493-46cb-8467-7bd5dd369c78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:43.807889 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc456c70-2493-46cb-8467-7bd5dd369c78 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:43.811704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc456c70-2493-46cb-8467-7bd5dd369c78 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:43.817643 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc456c70-2493-46cb-8467-7bd5dd369c78 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:43.909278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1039/2055] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:43 2022] POST /identity/v3/projects => generated 383 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:43.909470 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c44cc3a-6370-43fa-85e9-bf282b7edd7c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:43.921850 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c44cc3a-6370-43fa-85e9-bf282b7edd7c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:44.028256 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17683f28-5dcf-4cbf-a0d1-2327182bd5ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:44.029891 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17683f28-5dcf-4cbf-a0d1-2327182bd5ff admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:44.030286 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17683f28-5dcf-4cbf-a0d1-2327182bd5ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:44.030633 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17683f28-5dcf-4cbf-a0d1-2327182bd5ff admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:44.031347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17683f28-5dcf-4cbf-a0d1-2327182bd5ff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:44.033724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17683f28-5dcf-4cbf-a0d1-2327182bd5ff admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:44.048809 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:44.068928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1017/2056] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:43 2022] POST /identity/v3/users => generated 292 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:19:44.073568 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc456c70-2493-46cb-8467-7bd5dd369c78 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:44.090291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc456c70-2493-46cb-8467-7bd5dd369c78 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:44.100171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d96dfb0-2eed-410f-abd6-2e9d2f5f856b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:44.101218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d96dfb0-2eed-410f-abd6-2e9d2f5f856b admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:44.101691 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d96dfb0-2eed-410f-abd6-2e9d2f5f856b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:44.102371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d96dfb0-2eed-410f-abd6-2e9d2f5f856b admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:44.103366 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d96dfb0-2eed-410f-abd6-2e9d2f5f856b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:44.106984 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d96dfb0-2eed-410f-abd6-2e9d2f5f856b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:44.126712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1040/2057] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:44 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:44.132328 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17683f28-5dcf-4cbf-a0d1-2327182bd5ff admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:44.145934 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17683f28-5dcf-4cbf-a0d1-2327182bd5ff admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:44.152872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df2bd32-0494-4e16-8efa-042d2efb333a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:44.154025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df2bd32-0494-4e16-8efa-042d2efb333a admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:44.154575 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df2bd32-0494-4e16-8efa-042d2efb333a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:44.155127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df2bd32-0494-4e16-8efa-042d2efb333a admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:44.156390 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df2bd32-0494-4e16-8efa-042d2efb333a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:44.158329 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df2bd32-0494-4e16-8efa-042d2efb333a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:44.175680 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1018/2058] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:44 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:44.181192 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d96dfb0-2eed-410f-abd6-2e9d2f5f856b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:44.195079 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d96dfb0-2eed-410f-abd6-2e9d2f5f856b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:44.202803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ee9caa-8ff8-4320-858b-bd79b13abff7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:44.203859 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ee9caa-8ff8-4320-858b-bd79b13abff7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:44.204381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ee9caa-8ff8-4320-858b-bd79b13abff7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:44.204888 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ee9caa-8ff8-4320-858b-bd79b13abff7 admin admin] PATH_INFO: `/v3/projects/809269e9842048c1a8336e92c8d343d8/users/21d2b9969b204bbca036fb257589ab22/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:44.232184 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ee9caa-8ff8-4320-858b-bd79b13abff7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=809269e9842048c1a8336e92c8d343d8, user_id=21d2b9969b204bbca036fb257589ab22, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:44.236447 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ee9caa-8ff8-4320-858b-bd79b13abff7 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:44.274366 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1041/2059] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:44 2022] PUT /identity/v3/projects/809269e9842048c1a8336e92c8d343d8/users/21d2b9969b204bbca036fb257589ab22/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:44.289174 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21324d4-e8e4-4d0d-b4f5-f505032bd8f6 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:44.289782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21324d4-e8e4-4d0d-b4f5-f505032bd8f6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:44.290297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21324d4-e8e4-4d0d-b4f5-f505032bd8f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:44.309309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ee9caa-8ff8-4320-858b-bd79b13abff7 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:44.312247 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-20ee9caa-8ff8-4320-858b-bd79b13abff7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:44.395188 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b21324d4-e8e4-4d0d-b4f5-f505032bd8f6 None None] MFA Rules not processed for user `21d2b9969b204bbca036fb257589ab22`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:44.396735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b21324d4-e8e4-4d0d-b4f5-f505032bd8f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:44.510569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ee9caa-8ff8-4320-858b-bd79b13abff7 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:44.522368 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9162d1-1959-4995-bb00-ec1d009821a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:44.523060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9162d1-1959-4995-bb00-ec1d009821a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:44.523338 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9162d1-1959-4995-bb00-ec1d009821a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:44.523671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9162d1-1959-4995-bb00-ec1d009821a6 admin admin] PATH_INFO: `/v3/projects/b56b4625387541cfa4ebb589e7404204` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:44.529674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9162d1-1959-4995-bb00-ec1d009821a6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b56b4625387541cfa4ebb589e7404204)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:44.531637 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9162d1-1959-4995-bb00-ec1d009821a6 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:44.577155 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1019/2060] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:44 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 290 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:44.581557 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21324d4-e8e4-4d0d-b4f5-f505032bd8f6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:44.607092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21324d4-e8e4-4d0d-b4f5-f505032bd8f6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:44.620294 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df05af0e-17c8-4c98-bdc2-8c18e0522412 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:44.621193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df05af0e-17c8-4c98-bdc2-8c18e0522412 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:44.621682 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df05af0e-17c8-4c98-bdc2-8c18e0522412 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:44.622679 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df05af0e-17c8-4c98-bdc2-8c18e0522412 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:44.623612 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df05af0e-17c8-4c98-bdc2-8c18e0522412 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:44.626584 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df05af0e-17c8-4c98-bdc2-8c18e0522412 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:44.658024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0a9162d1-1959-4995-bb00-ec1d009821a6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b56b4625387541cfa4ebb589e7404204', 'request_id': 'req-0a9162d1-1959-4995-bb00-ec1d009821a6'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:44.672904 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0a9162d1-1959-4995-bb00-ec1d009821a6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b56b4625387541cfa4ebb589e7404204', 'request_id': 'req-0a9162d1-1959-4995-bb00-ec1d009821a6'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:44.688111 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1020/2061] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:44 2022] POST /identity/v3/projects => generated 385 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:44.692513 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df05af0e-17c8-4c98-bdc2-8c18e0522412 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:44.692666 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0a9162d1-1959-4995-bb00-ec1d009821a6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b56b4625387541cfa4ebb589e7404204', 'request_id': 'req-0a9162d1-1959-4995-bb00-ec1d009821a6'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:44.716341 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0a9162d1-1959-4995-bb00-ec1d009821a6 admin admin] The token cache is being invalidate because project b56b4625387541cfa4ebb589e7404204 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:44.716793 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0a9162d1-1959-4995-bb00-ec1d009821a6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:44.717488 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0a9162d1-1959-4995-bb00-ec1d009821a6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:44.722114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1042/2062] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:44 2022] DELETE /identity/v3/projects/b56b4625387541cfa4ebb589e7404204 => generated 0 bytes in 411 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:19:44.723123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9162d1-1959-4995-bb00-ec1d009821a6 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:44.725343 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0a9162d1-1959-4995-bb00-ec1d009821a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:44.790757 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:44.833679 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9162d1-1959-4995-bb00-ec1d009821a6 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:44.864898 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df05af0e-17c8-4c98-bdc2-8c18e0522412 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:44.873083 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a2797e-1f6c-4f55-a643-b342aaf89b9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:44.873721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a2797e-1f6c-4f55-a643-b342aaf89b9d admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:44.874007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a2797e-1f6c-4f55-a643-b342aaf89b9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:44.874312 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a2797e-1f6c-4f55-a643-b342aaf89b9d admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:44.875306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a2797e-1f6c-4f55-a643-b342aaf89b9d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:44.878022 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a2797e-1f6c-4f55-a643-b342aaf89b9d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:44.932104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1021/2063] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:44 2022] POST /identity/v3/users => generated 293 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:44.943608 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a2797e-1f6c-4f55-a643-b342aaf89b9d admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:44.953512 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-25a2797e-1f6c-4f55-a643-b342aaf89b9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:44.967414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142ffe00-a28c-4d91-84bc-e6c13161d7b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:44.968291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142ffe00-a28c-4d91-84bc-e6c13161d7b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:44.968977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142ffe00-a28c-4d91-84bc-e6c13161d7b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:44.969589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142ffe00-a28c-4d91-84bc-e6c13161d7b3 admin admin] PATH_INFO: `/v3/users/2ea3c6571635435b83e84e45b502e442` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:44.978191 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a2797e-1f6c-4f55-a643-b342aaf89b9d admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:44.984339 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142ffe00-a28c-4d91-84bc-e6c13161d7b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ea3c6571635435b83e84e45b502e442)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:44.990908 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142ffe00-a28c-4d91-84bc-e6c13161d7b3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:44.992780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfff92a-960d-4dca-b7aa-279688cc67bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:44.993608 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfff92a-960d-4dca-b7aa-279688cc67bc admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:44.994056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfff92a-960d-4dca-b7aa-279688cc67bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:44.994504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfff92a-960d-4dca-b7aa-279688cc67bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:44.995513 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfff92a-960d-4dca-b7aa-279688cc67bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:44.998322 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfff92a-960d-4dca-b7aa-279688cc67bc admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:45.021923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1022/2064] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:44 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:45.025616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfff92a-960d-4dca-b7aa-279688cc67bc admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:45.037818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfff92a-960d-4dca-b7aa-279688cc67bc admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:45.052271 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb72eae4-7dfe-4c1b-8a9a-d3edde545b19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:45.055290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb72eae4-7dfe-4c1b-8a9a-d3edde545b19 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:45.055593 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb72eae4-7dfe-4c1b-8a9a-d3edde545b19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:45.056093 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb72eae4-7dfe-4c1b-8a9a-d3edde545b19 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:45.056772 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb72eae4-7dfe-4c1b-8a9a-d3edde545b19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:45.060505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb72eae4-7dfe-4c1b-8a9a-d3edde545b19 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:45.086581 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1023/2065] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:45 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:45.087650 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb72eae4-7dfe-4c1b-8a9a-d3edde545b19 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:45.092786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-142ffe00-a28c-4d91-84bc-e6c13161d7b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ea3c6571635435b83e84e45b502e442'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:45.093905 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-142ffe00-a28c-4d91-84bc-e6c13161d7b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ea3c6571635435b83e84e45b502e442'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:45.100358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb72eae4-7dfe-4c1b-8a9a-d3edde545b19 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:45.104234 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-142ffe00-a28c-4d91-84bc-e6c13161d7b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ea3c6571635435b83e84e45b502e442'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:45.105297 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-142ffe00-a28c-4d91-84bc-e6c13161d7b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ea3c6571635435b83e84e45b502e442'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:45.106206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e129deb6-5a19-4d2b-9f15-66abfb1b682c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:45.107536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e129deb6-5a19-4d2b-9f15-66abfb1b682c admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:45.108358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e129deb6-5a19-4d2b-9f15-66abfb1b682c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:45.108358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e129deb6-5a19-4d2b-9f15-66abfb1b682c admin admin] PATH_INFO: `/v3/projects/9bc52069875c413195933c9143cc6ff6/users/6f31eaa4434949fa805db561232be9ac/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:45.132372 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e129deb6-5a19-4d2b-9f15-66abfb1b682c admin admin] RBAC: Authorizing `identity:create_grant(project_id=9bc52069875c413195933c9143cc6ff6, user_id=6f31eaa4434949fa805db561232be9ac, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:45.133173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-142ffe00-a28c-4d91-84bc-e6c13161d7b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ea3c6571635435b83e84e45b502e442'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:45.135066 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e129deb6-5a19-4d2b-9f15-66abfb1b682c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:45.139559 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:45.160635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1043/2066] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:44 2022] DELETE /identity/v3/users/2ea3c6571635435b83e84e45b502e442 => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:45.183551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1024/2067] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:45 2022] PUT /identity/v3/projects/9bc52069875c413195933c9143cc6ff6/users/6f31eaa4434949fa805db561232be9ac/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:45.190683 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b483dac-2e5c-4723-82a7-db775bcdfe69 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:45.191281 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b483dac-2e5c-4723-82a7-db775bcdfe69 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:45.191794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b483dac-2e5c-4723-82a7-db775bcdfe69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:45.229899 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-3b483dac-2e5c-4723-82a7-db775bcdfe69 None None] MFA Rules not processed for user `6f31eaa4434949fa805db561232be9ac`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:45.231045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3b483dac-2e5c-4723-82a7-db775bcdfe69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:45.293773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:45.331305 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1044/2068] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:45 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:45.332643 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e129deb6-5a19-4d2b-9f15-66abfb1b682c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:45.335960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e129deb6-5a19-4d2b-9f15-66abfb1b682c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:45.440158 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e129deb6-5a19-4d2b-9f15-66abfb1b682c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:45.485299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b483dac-2e5c-4723-82a7-db775bcdfe69 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:45.486809 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3b483dac-2e5c-4723-82a7-db775bcdfe69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:45.518640 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871a4564-2a4a-4605-bd11-f6737901fd60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:45.519369 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871a4564-2a4a-4605-bd11-f6737901fd60 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:45.520270 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871a4564-2a4a-4605-bd11-f6737901fd60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:45.520777 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871a4564-2a4a-4605-bd11-f6737901fd60 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:45.521878 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871a4564-2a4a-4605-bd11-f6737901fd60 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:45.523971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871a4564-2a4a-4605-bd11-f6737901fd60 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:45.568802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1025/2069] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:45 2022] POST /identity/v3/projects => generated 385 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:45.572706 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871a4564-2a4a-4605-bd11-f6737901fd60 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:45.630719 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b483dac-2e5c-4723-82a7-db775bcdfe69 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:45.686898 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871a4564-2a4a-4605-bd11-f6737901fd60 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:45.697029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec80b00-2da3-4d9e-ab7b-c97677822e6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:45.697621 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec80b00-2da3-4d9e-ab7b-c97677822e6e admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:45.697903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec80b00-2da3-4d9e-ab7b-c97677822e6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:45.698832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec80b00-2da3-4d9e-ab7b-c97677822e6e admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:45.700920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec80b00-2da3-4d9e-ab7b-c97677822e6e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:45.703674 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec80b00-2da3-4d9e-ab7b-c97677822e6e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:45.723890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ae5c36-5985-4b10-a446-68fafbc8ee58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:45.724602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ae5c36-5985-4b10-a446-68fafbc8ee58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:45.724922 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ae5c36-5985-4b10-a446-68fafbc8ee58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:45.725234 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ae5c36-5985-4b10-a446-68fafbc8ee58 admin admin] PATH_INFO: `/v3/projects/3f9277124ad14ae0be2ff58cd312dbda` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:45.735368 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ae5c36-5985-4b10-a446-68fafbc8ee58 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3f9277124ad14ae0be2ff58cd312dbda)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:45.736469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1026/2070] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:45 2022] POST /identity/v3/users => generated 299 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:45.737518 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ae5c36-5985-4b10-a446-68fafbc8ee58 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:45.741455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec80b00-2da3-4d9e-ab7b-c97677822e6e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:45.759996 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec80b00-2da3-4d9e-ab7b-c97677822e6e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:45.765515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73312bd9-115f-407c-9b79-9159c36f4ed4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:45.766463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73312bd9-115f-407c-9b79-9159c36f4ed4 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:45.766749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73312bd9-115f-407c-9b79-9159c36f4ed4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:45.767048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73312bd9-115f-407c-9b79-9159c36f4ed4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:45.767876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73312bd9-115f-407c-9b79-9159c36f4ed4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:45.769306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73312bd9-115f-407c-9b79-9159c36f4ed4 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:45.785591 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1027/2071] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:45 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:45.790107 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73312bd9-115f-407c-9b79-9159c36f4ed4 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:45.803375 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73312bd9-115f-407c-9b79-9159c36f4ed4 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:45.808814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:45.843809 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-99ae5c36-5985-4b10-a446-68fafbc8ee58 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3f9277124ad14ae0be2ff58cd312dbda', 'request_id': 'req-99ae5c36-5985-4b10-a446-68fafbc8ee58'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:45.856051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-99ae5c36-5985-4b10-a446-68fafbc8ee58 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3f9277124ad14ae0be2ff58cd312dbda', 'request_id': 'req-99ae5c36-5985-4b10-a446-68fafbc8ee58'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:45.869620 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-99ae5c36-5985-4b10-a446-68fafbc8ee58 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3f9277124ad14ae0be2ff58cd312dbda', 'request_id': 'req-99ae5c36-5985-4b10-a446-68fafbc8ee58'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:45.897631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-99ae5c36-5985-4b10-a446-68fafbc8ee58 admin admin] The token cache is being invalidate because project 3f9277124ad14ae0be2ff58cd312dbda was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:45.897923 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-99ae5c36-5985-4b10-a446-68fafbc8ee58 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:45.899344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-99ae5c36-5985-4b10-a446-68fafbc8ee58 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:45.905686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1045/2072] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:45 2022] DELETE /identity/v3/projects/3f9277124ad14ae0be2ff58cd312dbda => generated 0 bytes in 417 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:45.906772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ae5c36-5985-4b10-a446-68fafbc8ee58 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:45.910962 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-99ae5c36-5985-4b10-a446-68fafbc8ee58 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:45.915615 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecf50b3-0a3a-4f4b-b991-406702a15e4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:45.916428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecf50b3-0a3a-4f4b-b991-406702a15e4b admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:45.916792 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecf50b3-0a3a-4f4b-b991-406702a15e4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:45.917136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecf50b3-0a3a-4f4b-b991-406702a15e4b admin admin] PATH_INFO: `/v3/projects/d814167c18944a57bfeff3002b244347/users/4d964cf647a04116aac7b0776a888b38/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:45.925067 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ae5c36-5985-4b10-a446-68fafbc8ee58 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:45.930172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86564e1-8cdc-4d09-9de5-e9507cb964e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:45.930832 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86564e1-8cdc-4d09-9de5-e9507cb964e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:45.931154 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86564e1-8cdc-4d09-9de5-e9507cb964e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:45.931427 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86564e1-8cdc-4d09-9de5-e9507cb964e2 admin admin] PATH_INFO: `/v3/users/f40301242b1043f18563380ea9bc6eaf` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:45.932613 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86564e1-8cdc-4d09-9de5-e9507cb964e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f40301242b1043f18563380ea9bc6eaf)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:45.934729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86564e1-8cdc-4d09-9de5-e9507cb964e2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:45.945486 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecf50b3-0a3a-4f4b-b991-406702a15e4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=d814167c18944a57bfeff3002b244347, user_id=4d964cf647a04116aac7b0776a888b38, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:45.969243 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecf50b3-0a3a-4f4b-b991-406702a15e4b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:46.005623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1028/2073] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:45 2022] PUT /identity/v3/projects/d814167c18944a57bfeff3002b244347/users/4d964cf647a04116aac7b0776a888b38/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:46.007237 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecf50b3-0a3a-4f4b-b991-406702a15e4b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:46.012002 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fecf50b3-0a3a-4f4b-b991-406702a15e4b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:46.015684 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:46.092296 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c86564e1-8cdc-4d09-9de5-e9507cb964e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f40301242b1043f18563380ea9bc6eaf'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:46.112055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c86564e1-8cdc-4d09-9de5-e9507cb964e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f40301242b1043f18563380ea9bc6eaf'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:46.114343 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecf50b3-0a3a-4f4b-b991-406702a15e4b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:46.129940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fd6df0-4cea-465f-954c-e525bd45d78f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:46.130576 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fd6df0-4cea-465f-954c-e525bd45d78f admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:46.130875 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fd6df0-4cea-465f-954c-e525bd45d78f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:46.131209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fd6df0-4cea-465f-954c-e525bd45d78f admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:46.131860 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fd6df0-4cea-465f-954c-e525bd45d78f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:46.133757 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fd6df0-4cea-465f-954c-e525bd45d78f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:46.140024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c86564e1-8cdc-4d09-9de5-e9507cb964e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f40301242b1043f18563380ea9bc6eaf'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:46.141150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c86564e1-8cdc-4d09-9de5-e9507cb964e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f40301242b1043f18563380ea9bc6eaf'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:46.167827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c86564e1-8cdc-4d09-9de5-e9507cb964e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f40301242b1043f18563380ea9bc6eaf'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:46.182136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1029/2074] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:46 2022] GET /identity/v3/roles => generated 1420 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:19:46.183303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fd6df0-4cea-465f-954c-e525bd45d78f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:46.188452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-23fd6df0-4cea-465f-954c-e525bd45d78f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:46.216073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1046/2075] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:45 2022] DELETE /identity/v3/users/f40301242b1043f18563380ea9bc6eaf => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:46.225692 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fd6df0-4cea-465f-954c-e525bd45d78f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:46.316992 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114bd32c-d314-414b-b03b-ce2f1f896527 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:46.317589 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114bd32c-d314-414b-b03b-ce2f1f896527 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:46.317866 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114bd32c-d314-414b-b03b-ce2f1f896527 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:46.318136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114bd32c-d314-414b-b03b-ce2f1f896527 admin admin] PATH_INFO: `/v3/projects/d814167c18944a57bfeff3002b244347/users/4d964cf647a04116aac7b0776a888b38/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:46.320137 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114bd32c-d314-414b-b03b-ce2f1f896527 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d814167c18944a57bfeff3002b244347, user_id=4d964cf647a04116aac7b0776a888b38, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:46.323380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114bd32c-d314-414b-b03b-ce2f1f896527 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:46.359495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1030/2076] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:46 2022] PUT /identity/v3/projects/d814167c18944a57bfeff3002b244347/users/4d964cf647a04116aac7b0776a888b38/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:46.360643 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79470b2b-6c17-4e4e-9acd-d9cef3fd1e27 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:46.360960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79470b2b-6c17-4e4e-9acd-d9cef3fd1e27 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:46.361234 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79470b2b-6c17-4e4e-9acd-d9cef3fd1e27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:46.413530 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-79470b2b-6c17-4e4e-9acd-d9cef3fd1e27 None None] MFA Rules not processed for user `4d964cf647a04116aac7b0776a888b38`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:46.414469 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-79470b2b-6c17-4e4e-9acd-d9cef3fd1e27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:46.557683 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1047/2077] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:46 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:46.570209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114bd32c-d314-414b-b03b-ce2f1f896527 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:46.572252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-114bd32c-d314-414b-b03b-ce2f1f896527 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:46.616896 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:46.714246 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114bd32c-d314-414b-b03b-ce2f1f896527 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:46.719595 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c755eb42-06bf-4ac8-af58-bf5f505a89d7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:46.720423 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c755eb42-06bf-4ac8-af58-bf5f505a89d7 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:46.720845 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c755eb42-06bf-4ac8-af58-bf5f505a89d7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:46.721256 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c755eb42-06bf-4ac8-af58-bf5f505a89d7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:46.723502 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c755eb42-06bf-4ac8-af58-bf5f505a89d7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:46.876137 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c755eb42-06bf-4ac8-af58-bf5f505a89d7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:46.877528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c755eb42-06bf-4ac8-af58-bf5f505a89d7 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:46.910279 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1031/2078] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:46 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:47.122453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79470b2b-6c17-4e4e-9acd-d9cef3fd1e27 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:47.124574 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-79470b2b-6c17-4e4e-9acd-d9cef3fd1e27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:47.288995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79470b2b-6c17-4e4e-9acd-d9cef3fd1e27 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:47.297576 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bf600b-2512-42a5-a3a5-ba758506efcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:47.298317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bf600b-2512-42a5-a3a5-ba758506efcf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:47.299754 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bf600b-2512-42a5-a3a5-ba758506efcf admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:47.300036 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bf600b-2512-42a5-a3a5-ba758506efcf admin admin] PATH_INFO: `/v3/projects/e7dd7a401a434c81b1befe34ee1b3804` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:47.301676 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1bf600b-2512-42a5-a3a5-ba758506efcf admin admin] RBAC: Authorizing `identity:delete_project(project_id=e7dd7a401a434c81b1befe34ee1b3804)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:47.315640 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1bf600b-2512-42a5-a3a5-ba758506efcf admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:47.372001 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:47.436961 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a1bf600b-2512-42a5-a3a5-ba758506efcf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e7dd7a401a434c81b1befe34ee1b3804', 'request_id': 'req-a1bf600b-2512-42a5-a3a5-ba758506efcf'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:47.455331 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a1bf600b-2512-42a5-a3a5-ba758506efcf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e7dd7a401a434c81b1befe34ee1b3804', 'request_id': 'req-a1bf600b-2512-42a5-a3a5-ba758506efcf'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:47.468254 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a1bf600b-2512-42a5-a3a5-ba758506efcf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e7dd7a401a434c81b1befe34ee1b3804', 'request_id': 'req-a1bf600b-2512-42a5-a3a5-ba758506efcf'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:47.502112 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a1bf600b-2512-42a5-a3a5-ba758506efcf admin admin] The token cache is being invalidate because project e7dd7a401a434c81b1befe34ee1b3804 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:47.502424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a1bf600b-2512-42a5-a3a5-ba758506efcf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:47.503627 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a1bf600b-2512-42a5-a3a5-ba758506efcf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:47.505022 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1048/2079] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:47 2022] DELETE /identity/v3/projects/e7dd7a401a434c81b1befe34ee1b3804 => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:47.512083 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba26a09-a4af-4b77-89e0-77a5118b1a82 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:47.513414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba26a09-a4af-4b77-89e0-77a5118b1a82 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:47.513993 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba26a09-a4af-4b77-89e0-77a5118b1a82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:47.582423 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-fba26a09-a4af-4b77-89e0-77a5118b1a82 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:47.583837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fba26a09-a4af-4b77-89e0-77a5118b1a82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:47.673960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1032/2080] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:47.682356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29083f98-f71c-4b5d-898c-7a22dfb03bb6 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:47.682815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29083f98-f71c-4b5d-898c-7a22dfb03bb6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:47.683282 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29083f98-f71c-4b5d-898c-7a22dfb03bb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:47.720397 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-29083f98-f71c-4b5d-898c-7a22dfb03bb6 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:47.726941 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-29083f98-f71c-4b5d-898c-7a22dfb03bb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:47.748055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1049/2081] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:47.752008 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba26a09-a4af-4b77-89e0-77a5118b1a82 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:47.765880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba26a09-a4af-4b77-89e0-77a5118b1a82 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:47.782822 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fc5b11-0453-47dd-b66b-39dce965c405 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:47.783922 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fc5b11-0453-47dd-b66b-39dce965c405 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:47.784249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fc5b11-0453-47dd-b66b-39dce965c405 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:47.785015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fc5b11-0453-47dd-b66b-39dce965c405 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:47.785925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fc5b11-0453-47dd-b66b-39dce965c405 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:19:47.786939 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fc5b11-0453-47dd-b66b-39dce965c405 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:47.792875 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fc5b11-0453-47dd-b66b-39dce965c405 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:47.810019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1033/2082] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:19:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:19:47.814500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29083f98-f71c-4b5d-898c-7a22dfb03bb6 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:47.825798 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29083f98-f71c-4b5d-898c-7a22dfb03bb6 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:47.833238 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c69dd58-f361-48f4-bb8c-b4ee637fb989 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:47.834086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c69dd58-f361-48f4-bb8c-b4ee637fb989 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:47.834446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c69dd58-f361-48f4-bb8c-b4ee637fb989 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:47.834795 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c69dd58-f361-48f4-bb8c-b4ee637fb989 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:47.835711 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c69dd58-f361-48f4-bb8c-b4ee637fb989 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:47.837993 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c69dd58-f361-48f4-bb8c-b4ee637fb989 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:47.876383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1050/2083] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:47 2022] POST /identity/v3/projects => generated 393 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:47.881589 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fc5b11-0453-47dd-b66b-39dce965c405 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:47.898338 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fc5b11-0453-47dd-b66b-39dce965c405 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:47.991263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547773ae-d039-4323-b606-dd5e0e6c4301 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:47.992004 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547773ae-d039-4323-b606-dd5e0e6c4301 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:47.992358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547773ae-d039-4323-b606-dd5e0e6c4301 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:47.992705 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547773ae-d039-4323-b606-dd5e0e6c4301 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:47.993540 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547773ae-d039-4323-b606-dd5e0e6c4301 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:47.995973 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547773ae-d039-4323-b606-dd5e0e6c4301 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:48.016311 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:48.046903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1034/2084] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:47 2022] POST /identity/v3/users => generated 297 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:48.049920 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c69dd58-f361-48f4-bb8c-b4ee637fb989 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:48.067547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c69dd58-f361-48f4-bb8c-b4ee637fb989 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:48.073580 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b885e3d0-4d13-4ad6-8f21-efcfd17e9292 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:48.075097 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b885e3d0-4d13-4ad6-8f21-efcfd17e9292 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:48.076059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b885e3d0-4d13-4ad6-8f21-efcfd17e9292 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:48.076474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b885e3d0-4d13-4ad6-8f21-efcfd17e9292 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:48.077403 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b885e3d0-4d13-4ad6-8f21-efcfd17e9292 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:48.079217 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b885e3d0-4d13-4ad6-8f21-efcfd17e9292 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:48.095440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1051/2085] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:48 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:19:48.096280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547773ae-d039-4323-b606-dd5e0e6c4301 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:48.119068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547773ae-d039-4323-b606-dd5e0e6c4301 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:48.126730 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7bc60d1-8499-4009-be3b-069ebae7474b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:48.127570 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bc60d1-8499-4009-be3b-069ebae7474b admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:48.127861 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bc60d1-8499-4009-be3b-069ebae7474b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:48.128178 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bc60d1-8499-4009-be3b-069ebae7474b admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:48.128920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7bc60d1-8499-4009-be3b-069ebae7474b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:48.130639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7bc60d1-8499-4009-be3b-069ebae7474b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:48.151564 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1035/2086] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:48 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:48.159480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b885e3d0-4d13-4ad6-8f21-efcfd17e9292 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:48.171907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b885e3d0-4d13-4ad6-8f21-efcfd17e9292 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:48.177754 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4742827-1e85-4cbd-92eb-9907b6877794 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:48.179344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4742827-1e85-4cbd-92eb-9907b6877794 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:48.179344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4742827-1e85-4cbd-92eb-9907b6877794 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:48.179344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4742827-1e85-4cbd-92eb-9907b6877794 admin admin] PATH_INFO: `/v3/projects/23de74e9ab87420c8dfee0f2f7fe57c4/users/7a0261144c884a5a8fc320c19abbbe2b/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:48.195657 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4742827-1e85-4cbd-92eb-9907b6877794 admin admin] RBAC: Authorizing `identity:create_grant(project_id=23de74e9ab87420c8dfee0f2f7fe57c4, user_id=7a0261144c884a5a8fc320c19abbbe2b, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:48.198351 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4742827-1e85-4cbd-92eb-9907b6877794 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:48.214573 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1052/2087] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:48 2022] PUT /identity/v3/projects/23de74e9ab87420c8dfee0f2f7fe57c4/users/7a0261144c884a5a8fc320c19abbbe2b/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:19:48.227427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af64333c-2013-4ec8-a23b-a39b8cf10c51 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:48.227720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af64333c-2013-4ec8-a23b-a39b8cf10c51 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:48.228532 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af64333c-2013-4ec8-a23b-a39b8cf10c51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:48.259591 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-af64333c-2013-4ec8-a23b-a39b8cf10c51 None None] MFA Rules not processed for user `7a0261144c884a5a8fc320c19abbbe2b`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:48.260903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-af64333c-2013-4ec8-a23b-a39b8cf10c51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:48.362137 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1036/2088] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:48 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:48.367590 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4742827-1e85-4cbd-92eb-9907b6877794 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:48.493930 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4742827-1e85-4cbd-92eb-9907b6877794 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:48.501299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8871bb6-b83d-4272-837a-11c219391eaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:48.502423 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8871bb6-b83d-4272-837a-11c219391eaa admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:48.502795 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8871bb6-b83d-4272-837a-11c219391eaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:48.503190 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8871bb6-b83d-4272-837a-11c219391eaa admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:48.503939 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8871bb6-b83d-4272-837a-11c219391eaa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:48.506101 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8871bb6-b83d-4272-837a-11c219391eaa admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:48.565341 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af64333c-2013-4ec8-a23b-a39b8cf10c51 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:48.568022 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1053/2089] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:48 2022] POST /identity/v3/projects => generated 393 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:48.583068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af64333c-2013-4ec8-a23b-a39b8cf10c51 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:48.716501 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdd4f4c-3b5e-4e9e-a361-ad213e8bb70a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:48.718800 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdd4f4c-3b5e-4e9e-a361-ad213e8bb70a admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:48.719097 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdd4f4c-3b5e-4e9e-a361-ad213e8bb70a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:48.719382 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdd4f4c-3b5e-4e9e-a361-ad213e8bb70a admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:48.720014 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdd4f4c-3b5e-4e9e-a361-ad213e8bb70a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:48.722037 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdd4f4c-3b5e-4e9e-a361-ad213e8bb70a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:48.726571 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8871bb6-b83d-4272-837a-11c219391eaa admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:48.728644 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f8871bb6-b83d-4272-837a-11c219391eaa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:48.753885 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1037/2090] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:48 2022] POST /identity/v3/users => generated 303 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:48.755131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdd4f4c-3b5e-4e9e-a361-ad213e8bb70a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:48.769247 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdd4f4c-3b5e-4e9e-a361-ad213e8bb70a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:48.777533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5c3217-d06b-49fd-a64b-9b4fa13fa1b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:48.778159 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5c3217-d06b-49fd-a64b-9b4fa13fa1b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:48.778517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5c3217-d06b-49fd-a64b-9b4fa13fa1b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:48.778793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5c3217-d06b-49fd-a64b-9b4fa13fa1b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:48.779731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5c3217-d06b-49fd-a64b-9b4fa13fa1b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:48.781081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5c3217-d06b-49fd-a64b-9b4fa13fa1b7 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:48.786344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:48.798929 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1038/2091] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:48 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:19:48.803641 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5c3217-d06b-49fd-a64b-9b4fa13fa1b7 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:48.807066 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8871bb6-b83d-4272-837a-11c219391eaa admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:48.811438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6619b1d4-70a8-4a60-8678-4f6324b1c56c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:48.812174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6619b1d4-70a8-4a60-8678-4f6324b1c56c service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:48.813009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6619b1d4-70a8-4a60-8678-4f6324b1c56c service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:48.813009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6619b1d4-70a8-4a60-8678-4f6324b1c56c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:48.814975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6619b1d4-70a8-4a60-8678-4f6324b1c56c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:48.821322 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5c3217-d06b-49fd-a64b-9b4fa13fa1b7 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:48.828698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9723abf9-7c11-491e-98be-11b253be93a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:48.829514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9723abf9-7c11-491e-98be-11b253be93a8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:48.829993 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9723abf9-7c11-491e-98be-11b253be93a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:48.830257 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9723abf9-7c11-491e-98be-11b253be93a8 admin admin] PATH_INFO: `/v3/projects/eccc2cce5a7e44f9a5394ebb46fce162/users/4cf5a1f5ce0c417f9f099cc0bbf19050/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:48.851733 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9723abf9-7c11-491e-98be-11b253be93a8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eccc2cce5a7e44f9a5394ebb46fce162, user_id=4cf5a1f5ce0c417f9f099cc0bbf19050, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:48.854555 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9723abf9-7c11-491e-98be-11b253be93a8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:48.883882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1039/2092] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:48 2022] PUT /identity/v3/projects/eccc2cce5a7e44f9a5394ebb46fce162/users/4cf5a1f5ce0c417f9f099cc0bbf19050/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:48.885201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9723abf9-7c11-491e-98be-11b253be93a8 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:48.922372 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6619b1d4-70a8-4a60-8678-4f6324b1c56c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:48.925263 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6619b1d4-70a8-4a60-8678-4f6324b1c56c service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:48.943679 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:48.944907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1054/2093] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:48 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 219 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:19:49.024686 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9723abf9-7c11-491e-98be-11b253be93a8 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:49.031123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248e650d-ef93-4726-8b2f-d3d688d66cca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:49.032081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248e650d-ef93-4726-8b2f-d3d688d66cca admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:49.032423 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248e650d-ef93-4726-8b2f-d3d688d66cca admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:49.032696 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248e650d-ef93-4726-8b2f-d3d688d66cca admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:49.033337 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248e650d-ef93-4726-8b2f-d3d688d66cca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:49.035169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248e650d-ef93-4726-8b2f-d3d688d66cca admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:49.049872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1040/2094] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:48 2022] GET /identity/v3/roles => generated 1420 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:49.055122 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6619b1d4-70a8-4a60-8678-4f6324b1c56c service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:49.077232 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6619b1d4-70a8-4a60-8678-4f6324b1c56c service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:49.087467 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583629f8-bb03-4773-9ad8-256e4c13d69c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:49.088216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583629f8-bb03-4773-9ad8-256e4c13d69c admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:49.090244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583629f8-bb03-4773-9ad8-256e4c13d69c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:49.090244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583629f8-bb03-4773-9ad8-256e4c13d69c admin admin] PATH_INFO: `/v3/projects/eccc2cce5a7e44f9a5394ebb46fce162/users/4cf5a1f5ce0c417f9f099cc0bbf19050/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:49.096520 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583629f8-bb03-4773-9ad8-256e4c13d69c admin admin] RBAC: Authorizing `identity:create_grant(project_id=eccc2cce5a7e44f9a5394ebb46fce162, user_id=4cf5a1f5ce0c417f9f099cc0bbf19050, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:49.100169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583629f8-bb03-4773-9ad8-256e4c13d69c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:49.130144 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1055/2095] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:49 2022] PUT /identity/v3/projects/eccc2cce5a7e44f9a5394ebb46fce162/users/4cf5a1f5ce0c417f9f099cc0bbf19050/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:49.143397 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49e912e-f22b-4d2b-a61a-c075255f4a72 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:49.143813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49e912e-f22b-4d2b-a61a-c075255f4a72 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:49.144895 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49e912e-f22b-4d2b-a61a-c075255f4a72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:49.187096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e49e912e-f22b-4d2b-a61a-c075255f4a72 None None] MFA Rules not processed for user `4cf5a1f5ce0c417f9f099cc0bbf19050`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:49.188553 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e49e912e-f22b-4d2b-a61a-c075255f4a72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:49.311881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e361bb-0f91-453b-a279-5d62da8e1679 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:49.312091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e361bb-0f91-453b-a279-5d62da8e1679 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:49.312373 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e361bb-0f91-453b-a279-5d62da8e1679 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:49.349533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1041/2096] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:49 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:49.357494 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49e912e-f22b-4d2b-a61a-c075255f4a72 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:49.359397 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e49e912e-f22b-4d2b-a61a-c075255f4a72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:49.406568 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-63e361bb-0f91-453b-a279-5d62da8e1679 None None] MFA Rules not processed for user `85178dc2280c4a54970fc207d1746608`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:49.418069 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-63e361bb-0f91-453b-a279-5d62da8e1679 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:49.494130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49e912e-f22b-4d2b-a61a-c075255f4a72 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:49.500284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a181fb-6799-4c5a-bd2d-f83d6a059dd6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:49.500969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a181fb-6799-4c5a-bd2d-f83d6a059dd6 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:49.501344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a181fb-6799-4c5a-bd2d-f83d6a059dd6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:49.501685 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a181fb-6799-4c5a-bd2d-f83d6a059dd6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:49.530632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a181fb-6799-4c5a-bd2d-f83d6a059dd6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:49.535486 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a181fb-6799-4c5a-bd2d-f83d6a059dd6 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:49.559284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1042/2097] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:49 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:19:49.588761 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1056/2098] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:49 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 275 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:19:49.589204 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a181fb-6799-4c5a-bd2d-f83d6a059dd6 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:49.590781 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-71a181fb-6799-4c5a-bd2d-f83d6a059dd6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:49.642608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e361bb-0f91-453b-a279-5d62da8e1679 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:49.663198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e361bb-0f91-453b-a279-5d62da8e1679 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:49.674601 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23956daf-9b6b-4d0e-b353-d1020398479e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:49.675569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23956daf-9b6b-4d0e-b353-d1020398479e service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:49.678054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23956daf-9b6b-4d0e-b353-d1020398479e service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:49.679129 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23956daf-9b6b-4d0e-b353-d1020398479e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:49.691622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:49.739360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a181fb-6799-4c5a-bd2d-f83d6a059dd6 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:49.751392 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d146061-ef9a-40e0-94be-50acde48b1ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:49.752031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d146061-ef9a-40e0-94be-50acde48b1ff admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:49.752640 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d146061-ef9a-40e0-94be-50acde48b1ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:49.752796 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d146061-ef9a-40e0-94be-50acde48b1ff admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:49.755915 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d146061-ef9a-40e0-94be-50acde48b1ff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:49.764449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d146061-ef9a-40e0-94be-50acde48b1ff admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:49.777285 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23956daf-9b6b-4d0e-b353-d1020398479e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:49.779644 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23956daf-9b6b-4d0e-b353-d1020398479e service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:49.796742 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1057/2099] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:49 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:19:49.826649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1043/2100] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:49 2022] POST /identity/v3/projects => generated 371 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:49.831331 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23956daf-9b6b-4d0e-b353-d1020398479e service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:49.846939 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23956daf-9b6b-4d0e-b353-d1020398479e service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:49.852093 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:50.000902 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6741d9d3-5f4e-4dc6-a76a-cb1fec6baadb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:50.002385 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6741d9d3-5f4e-4dc6-a76a-cb1fec6baadb admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:50.003001 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6741d9d3-5f4e-4dc6-a76a-cb1fec6baadb admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:50.003386 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6741d9d3-5f4e-4dc6-a76a-cb1fec6baadb admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:50.004134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6741d9d3-5f4e-4dc6-a76a-cb1fec6baadb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:50.009808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6741d9d3-5f4e-4dc6-a76a-cb1fec6baadb admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:50.020018 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d146061-ef9a-40e0-94be-50acde48b1ff admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:50.062552 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1058/2101] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:49 2022] POST /identity/v3/users => generated 292 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:50.064660 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6741d9d3-5f4e-4dc6-a76a-cb1fec6baadb admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:50.094702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6741d9d3-5f4e-4dc6-a76a-cb1fec6baadb admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:50.102720 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d40d1c-09d0-47b1-8008-9b089f7ec60f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:50.103339 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d146061-ef9a-40e0-94be-50acde48b1ff admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:50.103595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d40d1c-09d0-47b1-8008-9b089f7ec60f admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:50.104058 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d40d1c-09d0-47b1-8008-9b089f7ec60f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:50.104493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d40d1c-09d0-47b1-8008-9b089f7ec60f admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:50.106020 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d40d1c-09d0-47b1-8008-9b089f7ec60f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:50.107608 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba220c1-880e-4914-8346-a3e2e0d6c4a4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:50.108268 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba220c1-880e-4914-8346-a3e2e0d6c4a4 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:50.108544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba220c1-880e-4914-8346-a3e2e0d6c4a4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:50.108818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba220c1-880e-4914-8346-a3e2e0d6c4a4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:50.108943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d40d1c-09d0-47b1-8008-9b089f7ec60f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:50.110605 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fba220c1-880e-4914-8346-a3e2e0d6c4a4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:50.124428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1059/2102] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:50 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:50.129438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d40d1c-09d0-47b1-8008-9b089f7ec60f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:50.144955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d40d1c-09d0-47b1-8008-9b089f7ec60f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:50.155404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcce7300-a244-4c04-9f31-b8a82c140388 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:50.157786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcce7300-a244-4c04-9f31-b8a82c140388 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:50.158801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcce7300-a244-4c04-9f31-b8a82c140388 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:50.160422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcce7300-a244-4c04-9f31-b8a82c140388 admin admin] PATH_INFO: `/v3/projects/bf6e7bbb274340ecbc5025be563f7524/users/ee8f5c0bb6c947dfbd35022ecfac08c4/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:50.208319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcce7300-a244-4c04-9f31-b8a82c140388 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bf6e7bbb274340ecbc5025be563f7524, user_id=ee8f5c0bb6c947dfbd35022ecfac08c4, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:50.211003 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcce7300-a244-4c04-9f31-b8a82c140388 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:50.251869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba220c1-880e-4914-8346-a3e2e0d6c4a4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:50.252843 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1060/2103] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:50 2022] PUT /identity/v3/projects/bf6e7bbb274340ecbc5025be563f7524/users/ee8f5c0bb6c947dfbd35022ecfac08c4/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:50.253786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcce7300-a244-4c04-9f31-b8a82c140388 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:50.258284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba220c1-880e-4914-8346-a3e2e0d6c4a4 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:50.285199 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1044/2104] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:50 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 265 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:50.431978 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcce7300-a244-4c04-9f31-b8a82c140388 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:50.453077 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0569c427-0cf4-413e-b710-0897cf27f7b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:50.453806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0569c427-0cf4-413e-b710-0897cf27f7b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:50.454517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0569c427-0cf4-413e-b710-0897cf27f7b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:50.454796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0569c427-0cf4-413e-b710-0897cf27f7b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:50.455491 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0569c427-0cf4-413e-b710-0897cf27f7b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:50.457628 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0569c427-0cf4-413e-b710-0897cf27f7b9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:50.493824 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1061/2105] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:50 2022] GET /identity/v3/roles => generated 1420 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:50.494571 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba220c1-880e-4914-8346-a3e2e0d6c4a4 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:50.506195 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba220c1-880e-4914-8346-a3e2e0d6c4a4 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:50.511127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5e967d-cbde-415b-b799-f9af39d79519 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:50.511835 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5e967d-cbde-415b-b799-f9af39d79519 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:50.512115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5e967d-cbde-415b-b799-f9af39d79519 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:50.512389 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5e967d-cbde-415b-b799-f9af39d79519 admin admin] PATH_INFO: `/v3/projects/bf6e7bbb274340ecbc5025be563f7524/users/ee8f5c0bb6c947dfbd35022ecfac08c4/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:50.514879 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5e967d-cbde-415b-b799-f9af39d79519 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bf6e7bbb274340ecbc5025be563f7524, user_id=ee8f5c0bb6c947dfbd35022ecfac08c4, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:50.517771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5e967d-cbde-415b-b799-f9af39d79519 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:50.535726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1045/2106] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:50 2022] PUT /identity/v3/projects/bf6e7bbb274340ecbc5025be563f7524/users/ee8f5c0bb6c947dfbd35022ecfac08c4/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:53.972567 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0569c427-0cf4-413e-b710-0897cf27f7b9 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:54.002012 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0569c427-0cf4-413e-b710-0897cf27f7b9 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:54.148415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:54.163315 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91470f4c-19f2-4ffb-a601-1d6f3ae597a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:54.164224 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91470f4c-19f2-4ffb-a601-1d6f3ae597a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:54.165537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91470f4c-19f2-4ffb-a601-1d6f3ae597a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:54.166040 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91470f4c-19f2-4ffb-a601-1d6f3ae597a0 admin admin] PATH_INFO: `/v3/users/7a0261144c884a5a8fc320c19abbbe2b` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:54.169433 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91470f4c-19f2-4ffb-a601-1d6f3ae597a0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7a0261144c884a5a8fc320c19abbbe2b)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:54.176460 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91470f4c-19f2-4ffb-a601-1d6f3ae597a0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:54.267939 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-91470f4c-19f2-4ffb-a601-1d6f3ae597a0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7a0261144c884a5a8fc320c19abbbe2b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:54.268818 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-91470f4c-19f2-4ffb-a601-1d6f3ae597a0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7a0261144c884a5a8fc320c19abbbe2b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:54.280726 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-91470f4c-19f2-4ffb-a601-1d6f3ae597a0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7a0261144c884a5a8fc320c19abbbe2b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:54.281892 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-91470f4c-19f2-4ffb-a601-1d6f3ae597a0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7a0261144c884a5a8fc320c19abbbe2b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:54.301161 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-91470f4c-19f2-4ffb-a601-1d6f3ae597a0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7a0261144c884a5a8fc320c19abbbe2b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:54.324860 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1062/2107] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:53 2022] DELETE /identity/v3/users/7a0261144c884a5a8fc320c19abbbe2b => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:54.330226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578c8320-90fa-418b-be53-14ad0de5e71a None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:54.330734 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578c8320-90fa-418b-be53-14ad0de5e71a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:54.330811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578c8320-90fa-418b-be53-14ad0de5e71a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:54.373958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-578c8320-90fa-418b-be53-14ad0de5e71a None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:54.375905 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-578c8320-90fa-418b-be53-14ad0de5e71a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:54.474166 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1046/2108] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:54.485885 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91470f4c-19f2-4ffb-a601-1d6f3ae597a0 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:54.489319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-91470f4c-19f2-4ffb-a601-1d6f3ae597a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:54.591342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91470f4c-19f2-4ffb-a601-1d6f3ae597a0 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:54.595682 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ba0362-de1d-4d1f-a7d1-d9eb981b5fba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:54.596204 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ba0362-de1d-4d1f-a7d1-d9eb981b5fba service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:54.596547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ba0362-de1d-4d1f-a7d1-d9eb981b5fba service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:54.596885 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ba0362-de1d-4d1f-a7d1-d9eb981b5fba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:54.614544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ba0362-de1d-4d1f-a7d1-d9eb981b5fba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:54.616461 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ba0362-de1d-4d1f-a7d1-d9eb981b5fba service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:54.658662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1063/2109] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:54.988633 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578c8320-90fa-418b-be53-14ad0de5e71a None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:54.991280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-578c8320-90fa-418b-be53-14ad0de5e71a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:55.005688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578c8320-90fa-418b-be53-14ad0de5e71a None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:55.013174 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5050fc5d-91d2-40f6-a42f-a6f7774c5216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:55.014212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5050fc5d-91d2-40f6-a42f-a6f7774c5216 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:55.014694 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5050fc5d-91d2-40f6-a42f-a6f7774c5216 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:55.015232 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5050fc5d-91d2-40f6-a42f-a6f7774c5216 admin admin] PATH_INFO: `/v3/users/21d2b9969b204bbca036fb257589ab22` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:55.016690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5050fc5d-91d2-40f6-a42f-a6f7774c5216 admin admin] RBAC: Authorizing `identity:delete_user(user_id=21d2b9969b204bbca036fb257589ab22)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:55.019030 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5050fc5d-91d2-40f6-a42f-a6f7774c5216 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:55.027261 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:55.108313 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5050fc5d-91d2-40f6-a42f-a6f7774c5216 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21d2b9969b204bbca036fb257589ab22'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:55.109523 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5050fc5d-91d2-40f6-a42f-a6f7774c5216 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21d2b9969b204bbca036fb257589ab22'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:55.120612 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5050fc5d-91d2-40f6-a42f-a6f7774c5216 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21d2b9969b204bbca036fb257589ab22'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:55.122380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5050fc5d-91d2-40f6-a42f-a6f7774c5216 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21d2b9969b204bbca036fb257589ab22'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:55.156638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5050fc5d-91d2-40f6-a42f-a6f7774c5216 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21d2b9969b204bbca036fb257589ab22'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:55.201017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1047/2110] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:54 2022] DELETE /identity/v3/users/21d2b9969b204bbca036fb257589ab22 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:55.201646 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d52b81e-a595-4c7b-b047-493c4b149d08 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:55.202505 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d52b81e-a595-4c7b-b047-493c4b149d08 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:55.202787 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d52b81e-a595-4c7b-b047-493c4b149d08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:55.278522 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9d52b81e-a595-4c7b-b047-493c4b149d08 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:55.284476 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9d52b81e-a595-4c7b-b047-493c4b149d08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:55.420819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1064/2111] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:55.432225 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5050fc5d-91d2-40f6-a42f-a6f7774c5216 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:55.433737 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5050fc5d-91d2-40f6-a42f-a6f7774c5216 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:55.465123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d52b81e-a595-4c7b-b047-493c4b149d08 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:55.470215 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9d52b81e-a595-4c7b-b047-493c4b149d08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:55.494884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d52b81e-a595-4c7b-b047-493c4b149d08 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:55.504148 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49ce4e8-1e3a-490d-8724-2ab296db922f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:55.505538 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ce4e8-1e3a-490d-8724-2ab296db922f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:55.505823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ce4e8-1e3a-490d-8724-2ab296db922f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:55.506101 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ce4e8-1e3a-490d-8724-2ab296db922f admin admin] PATH_INFO: `/v3/projects/23de74e9ab87420c8dfee0f2f7fe57c4` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:55.507622 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49ce4e8-1e3a-490d-8724-2ab296db922f admin admin] RBAC: Authorizing `identity:delete_project(project_id=23de74e9ab87420c8dfee0f2f7fe57c4)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:55.509618 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49ce4e8-1e3a-490d-8724-2ab296db922f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:55.536826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:55.620000 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5050fc5d-91d2-40f6-a42f-a6f7774c5216 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:55.622729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d49ce4e8-1e3a-490d-8724-2ab296db922f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '23de74e9ab87420c8dfee0f2f7fe57c4', 'request_id': 'req-d49ce4e8-1e3a-490d-8724-2ab296db922f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:55.630699 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d49ce4e8-1e3a-490d-8724-2ab296db922f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '23de74e9ab87420c8dfee0f2f7fe57c4', 'request_id': 'req-d49ce4e8-1e3a-490d-8724-2ab296db922f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:55.643224 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d49ce4e8-1e3a-490d-8724-2ab296db922f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '23de74e9ab87420c8dfee0f2f7fe57c4', 'request_id': 'req-d49ce4e8-1e3a-490d-8724-2ab296db922f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:55.667709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d49ce4e8-1e3a-490d-8724-2ab296db922f admin admin] The token cache is being invalidate because project 23de74e9ab87420c8dfee0f2f7fe57c4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:55.669598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d49ce4e8-1e3a-490d-8724-2ab296db922f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:55.671772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d49ce4e8-1e3a-490d-8724-2ab296db922f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:55.676276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1065/2112] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:55 2022] DELETE /identity/v3/projects/23de74e9ab87420c8dfee0f2f7fe57c4 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:55.679945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49ce4e8-1e3a-490d-8724-2ab296db922f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:55.684935 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d49ce4e8-1e3a-490d-8724-2ab296db922f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:55.718711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831083bc-802e-46ed-86c6-9867a97e6ef5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:55.719375 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831083bc-802e-46ed-86c6-9867a97e6ef5 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:55.719653 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831083bc-802e-46ed-86c6-9867a97e6ef5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:55.719923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831083bc-802e-46ed-86c6-9867a97e6ef5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:55.722143 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-831083bc-802e-46ed-86c6-9867a97e6ef5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:55.800745 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:55.820445 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49ce4e8-1e3a-490d-8724-2ab296db922f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:55.829419 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3369fa-9b3b-4958-8772-db37a1444a49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:55.830379 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831083bc-802e-46ed-86c6-9867a97e6ef5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:55.830565 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3369fa-9b3b-4958-8772-db37a1444a49 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:55.830996 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3369fa-9b3b-4958-8772-db37a1444a49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:55.831344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3369fa-9b3b-4958-8772-db37a1444a49 admin admin] PATH_INFO: `/v3/users/4cf5a1f5ce0c417f9f099cc0bbf19050` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:55.832903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec3369fa-9b3b-4958-8772-db37a1444a49 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4cf5a1f5ce0c417f9f099cc0bbf19050)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:55.833473 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831083bc-802e-46ed-86c6-9867a97e6ef5 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:55.835353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec3369fa-9b3b-4958-8772-db37a1444a49 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:55.858020 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1048/2113] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:55.940050 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ec3369fa-9b3b-4958-8772-db37a1444a49 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4cf5a1f5ce0c417f9f099cc0bbf19050'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:55.943512 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ec3369fa-9b3b-4958-8772-db37a1444a49 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4cf5a1f5ce0c417f9f099cc0bbf19050'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:55.955874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ec3369fa-9b3b-4958-8772-db37a1444a49 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4cf5a1f5ce0c417f9f099cc0bbf19050'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:55.956605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ec3369fa-9b3b-4958-8772-db37a1444a49 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4cf5a1f5ce0c417f9f099cc0bbf19050'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:55.991269 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ec3369fa-9b3b-4958-8772-db37a1444a49 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4cf5a1f5ce0c417f9f099cc0bbf19050'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:56.024110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1066/2114] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:55 2022] DELETE /identity/v3/users/4cf5a1f5ce0c417f9f099cc0bbf19050 => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:19:56.608789 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831083bc-802e-46ed-86c6-9867a97e6ef5 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:56.622616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831083bc-802e-46ed-86c6-9867a97e6ef5 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:56.735734 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a866ecb-fabd-477d-b5c8-8154a5bac822 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:56.736453 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a866ecb-fabd-477d-b5c8-8154a5bac822 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:56.736755 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a866ecb-fabd-477d-b5c8-8154a5bac822 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:56.737056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a866ecb-fabd-477d-b5c8-8154a5bac822 admin admin] PATH_INFO: `/v3/projects/eccc2cce5a7e44f9a5394ebb46fce162` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:56.739692 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a866ecb-fabd-477d-b5c8-8154a5bac822 admin admin] RBAC: Authorizing `identity:delete_project(project_id=eccc2cce5a7e44f9a5394ebb46fce162)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:56.742007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a866ecb-fabd-477d-b5c8-8154a5bac822 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:56.861811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4a866ecb-fabd-477d-b5c8-8154a5bac822 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eccc2cce5a7e44f9a5394ebb46fce162', 'request_id': 'req-4a866ecb-fabd-477d-b5c8-8154a5bac822'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:56.872288 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4a866ecb-fabd-477d-b5c8-8154a5bac822 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eccc2cce5a7e44f9a5394ebb46fce162', 'request_id': 'req-4a866ecb-fabd-477d-b5c8-8154a5bac822'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:56.881102 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4a866ecb-fabd-477d-b5c8-8154a5bac822 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eccc2cce5a7e44f9a5394ebb46fce162', 'request_id': 'req-4a866ecb-fabd-477d-b5c8-8154a5bac822'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:56.909378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4a866ecb-fabd-477d-b5c8-8154a5bac822 admin admin] The token cache is being invalidate because project eccc2cce5a7e44f9a5394ebb46fce162 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:56.910030 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4a866ecb-fabd-477d-b5c8-8154a5bac822 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:56.914379 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4a866ecb-fabd-477d-b5c8-8154a5bac822 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:56.920912 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1049/2115] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:56 2022] DELETE /identity/v3/projects/eccc2cce5a7e44f9a5394ebb46fce162 => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:56.924307 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3369fa-9b3b-4958-8772-db37a1444a49 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:56.926719 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ec3369fa-9b3b-4958-8772-db37a1444a49 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:56.927649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd48158d-6ecf-419f-855a-24b96277d806 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:56.927955 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd48158d-6ecf-419f-855a-24b96277d806 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:56.928212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd48158d-6ecf-419f-855a-24b96277d806 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:56.955735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3369fa-9b3b-4958-8772-db37a1444a49 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:56.972860 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-bd48158d-6ecf-419f-855a-24b96277d806 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:56.975926 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-bd48158d-6ecf-419f-855a-24b96277d806 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:56.991785 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:57.122419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1050/2116] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:19:57.124350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6540ab-9873-466b-a4e2-a66d38cf4933 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:57.124826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6540ab-9873-466b-a4e2-a66d38cf4933 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:57.125130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6540ab-9873-466b-a4e2-a66d38cf4933 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:57.142748 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651b6024-c9f3-4147-a742-c78c12e8a683 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:57.143519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651b6024-c9f3-4147-a742-c78c12e8a683 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:57.143798 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651b6024-c9f3-4147-a742-c78c12e8a683 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:57.144072 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651b6024-c9f3-4147-a742-c78c12e8a683 admin admin] PATH_INFO: `/v3/projects/809269e9842048c1a8336e92c8d343d8` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:57.148252 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651b6024-c9f3-4147-a742-c78c12e8a683 admin admin] RBAC: Authorizing `identity:delete_project(project_id=809269e9842048c1a8336e92c8d343d8)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:57.150198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651b6024-c9f3-4147-a742-c78c12e8a683 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:57.173764 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-df6540ab-9873-466b-a4e2-a66d38cf4933 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:57.175746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-df6540ab-9873-466b-a4e2-a66d38cf4933 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:57.197002 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1051/2117] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:57.202901 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6540ab-9873-466b-a4e2-a66d38cf4933 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:57.222976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6540ab-9873-466b-a4e2-a66d38cf4933 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:57.242827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0113641e-5ac1-4b12-8eab-30534f99174e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:57.244109 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0113641e-5ac1-4b12-8eab-30534f99174e admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:57.244530 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0113641e-5ac1-4b12-8eab-30534f99174e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:57.244956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0113641e-5ac1-4b12-8eab-30534f99174e admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:57.245707 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0113641e-5ac1-4b12-8eab-30534f99174e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:19:57.246138 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0113641e-5ac1-4b12-8eab-30534f99174e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:57.249888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0113641e-5ac1-4b12-8eab-30534f99174e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:57.258636 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-651b6024-c9f3-4147-a742-c78c12e8a683 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '809269e9842048c1a8336e92c8d343d8', 'request_id': 'req-651b6024-c9f3-4147-a742-c78c12e8a683'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:57.268301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-651b6024-c9f3-4147-a742-c78c12e8a683 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '809269e9842048c1a8336e92c8d343d8', 'request_id': 'req-651b6024-c9f3-4147-a742-c78c12e8a683'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:57.268746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:57.280230 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-651b6024-c9f3-4147-a742-c78c12e8a683 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '809269e9842048c1a8336e92c8d343d8', 'request_id': 'req-651b6024-c9f3-4147-a742-c78c12e8a683'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:57.283516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1052/2118] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:19:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:19:57.284437 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0113641e-5ac1-4b12-8eab-30534f99174e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:57.312468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0113641e-5ac1-4b12-8eab-30534f99174e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:57.317051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-651b6024-c9f3-4147-a742-c78c12e8a683 admin admin] The token cache is being invalidate because project 809269e9842048c1a8336e92c8d343d8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:57.317342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-651b6024-c9f3-4147-a742-c78c12e8a683 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:57.318018 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-651b6024-c9f3-4147-a742-c78c12e8a683 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:57.319589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1067/2119] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:56 2022] DELETE /identity/v3/projects/809269e9842048c1a8336e92c8d343d8 => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:57.323435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651b6024-c9f3-4147-a742-c78c12e8a683 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:57.325022 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-651b6024-c9f3-4147-a742-c78c12e8a683 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:57.421499 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb25b0f3-faad-4577-b398-58a3920f34fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:57.422143 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb25b0f3-faad-4577-b398-58a3920f34fc admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:57.422420 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb25b0f3-faad-4577-b398-58a3920f34fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:57.422691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb25b0f3-faad-4577-b398-58a3920f34fc admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:57.423363 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb25b0f3-faad-4577-b398-58a3920f34fc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:57.427068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb25b0f3-faad-4577-b398-58a3920f34fc admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:57.463623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651b6024-c9f3-4147-a742-c78c12e8a683 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:57.470321 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5402726a-87a9-4181-87c3-78aa44b8294a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:57.471124 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5402726a-87a9-4181-87c3-78aa44b8294a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:57.471652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5402726a-87a9-4181-87c3-78aa44b8294a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:57.472158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5402726a-87a9-4181-87c3-78aa44b8294a admin admin] PATH_INFO: `/v3/users/6f31eaa4434949fa805db561232be9ac` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:57.473636 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5402726a-87a9-4181-87c3-78aa44b8294a admin admin] RBAC: Authorizing `identity:delete_user(user_id=6f31eaa4434949fa805db561232be9ac)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:57.477178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5402726a-87a9-4181-87c3-78aa44b8294a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:57.509640 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1053/2120] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:57 2022] POST /identity/v3/projects => generated 367 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:57.513530 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb25b0f3-faad-4577-b398-58a3920f34fc admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:57.517208 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-eb25b0f3-faad-4577-b398-58a3920f34fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:57.601045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5402726a-87a9-4181-87c3-78aa44b8294a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6f31eaa4434949fa805db561232be9ac'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:57.603163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5402726a-87a9-4181-87c3-78aa44b8294a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6f31eaa4434949fa805db561232be9ac'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:57.613225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5402726a-87a9-4181-87c3-78aa44b8294a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6f31eaa4434949fa805db561232be9ac'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:57.614002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5402726a-87a9-4181-87c3-78aa44b8294a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6f31eaa4434949fa805db561232be9ac'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:57.615534 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:57.642192 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5402726a-87a9-4181-87c3-78aa44b8294a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6f31eaa4434949fa805db561232be9ac'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:57.659484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1068/2121] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:57 2022] DELETE /identity/v3/users/6f31eaa4434949fa805db561232be9ac => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:19:57.681385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb25b0f3-faad-4577-b398-58a3920f34fc admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:57.820818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef107cd0-f61a-4b60-8193-5ee3430346bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:57.822431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef107cd0-f61a-4b60-8193-5ee3430346bc admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:57.823041 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef107cd0-f61a-4b60-8193-5ee3430346bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:57.823466 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef107cd0-f61a-4b60-8193-5ee3430346bc admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:57.824430 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef107cd0-f61a-4b60-8193-5ee3430346bc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:57.829434 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef107cd0-f61a-4b60-8193-5ee3430346bc admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:57.894991 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1054/2122] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:57 2022] POST /identity/v3/users => generated 284 bytes in 382 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:19:57.897850 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5402726a-87a9-4181-87c3-78aa44b8294a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:57.899299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5402726a-87a9-4181-87c3-78aa44b8294a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:57.921201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5402726a-87a9-4181-87c3-78aa44b8294a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:57.929102 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec5b62e-e5b1-446c-96da-e250d6f89861 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:57.931162 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec5b62e-e5b1-446c-96da-e250d6f89861 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:57.932101 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec5b62e-e5b1-446c-96da-e250d6f89861 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:57.932693 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec5b62e-e5b1-446c-96da-e250d6f89861 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:57.933916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec5b62e-e5b1-446c-96da-e250d6f89861 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:57.938817 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec5b62e-e5b1-446c-96da-e250d6f89861 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:57.961859 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1069/2123] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:57 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:57.965989 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef107cd0-f61a-4b60-8193-5ee3430346bc admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:57.985260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef107cd0-f61a-4b60-8193-5ee3430346bc admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:57.994179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86bcc10c-2299-4651-a08d-469bda2c2fec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:57.997297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bcc10c-2299-4651-a08d-469bda2c2fec admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:57.997843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bcc10c-2299-4651-a08d-469bda2c2fec admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:57.998431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bcc10c-2299-4651-a08d-469bda2c2fec admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:57.999439 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86bcc10c-2299-4651-a08d-469bda2c2fec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:58.001754 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86bcc10c-2299-4651-a08d-469bda2c2fec admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:58.017044 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1055/2124] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:57 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:58.021093 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec5b62e-e5b1-446c-96da-e250d6f89861 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:58.039697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec5b62e-e5b1-446c-96da-e250d6f89861 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:58.053437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2a925d-647a-4eb9-a282-5a9de91b9185 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:58.054500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2a925d-647a-4eb9-a282-5a9de91b9185 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:58.054926 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2a925d-647a-4eb9-a282-5a9de91b9185 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:58.055299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2a925d-647a-4eb9-a282-5a9de91b9185 admin admin] PATH_INFO: `/v3/projects/1fe54ba6fcc6433fb46a3afec1760029/users/14a6ec692df645c7878073d04fd97dd8/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:58.064510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86bcc10c-2299-4651-a08d-469bda2c2fec admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:58.066988 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-86bcc10c-2299-4651-a08d-469bda2c2fec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:58.093012 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86bcc10c-2299-4651-a08d-469bda2c2fec admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:58.102762 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2a925d-647a-4eb9-a282-5a9de91b9185 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1fe54ba6fcc6433fb46a3afec1760029, user_id=14a6ec692df645c7878073d04fd97dd8, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:58.106326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2a925d-647a-4eb9-a282-5a9de91b9185 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:58.107729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e92045d-d756-41a0-9d01-1a3f09d552ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:58.108302 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e92045d-d756-41a0-9d01-1a3f09d552ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:58.108695 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e92045d-d756-41a0-9d01-1a3f09d552ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:58.109094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e92045d-d756-41a0-9d01-1a3f09d552ad admin admin] PATH_INFO: `/v3/projects/9bc52069875c413195933c9143cc6ff6` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:58.111602 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e92045d-d756-41a0-9d01-1a3f09d552ad admin admin] RBAC: Authorizing `identity:delete_project(project_id=9bc52069875c413195933c9143cc6ff6)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:58.119742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e92045d-d756-41a0-9d01-1a3f09d552ad admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:58.139094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:58.145977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1070/2125] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:58 2022] PUT /identity/v3/projects/1fe54ba6fcc6433fb46a3afec1760029/users/14a6ec692df645c7878073d04fd97dd8/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:58.157116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dc86d4-69d7-4722-aeb8-981b415cc9d6 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:58.157863 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dc86d4-69d7-4722-aeb8-981b415cc9d6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:58.158096 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dc86d4-69d7-4722-aeb8-981b415cc9d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:58.210020 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0e92045d-d756-41a0-9d01-1a3f09d552ad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9bc52069875c413195933c9143cc6ff6', 'request_id': 'req-0e92045d-d756-41a0-9d01-1a3f09d552ad'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:58.219401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0e92045d-d756-41a0-9d01-1a3f09d552ad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9bc52069875c413195933c9143cc6ff6', 'request_id': 'req-0e92045d-d756-41a0-9d01-1a3f09d552ad'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:58.222917 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-06dc86d4-69d7-4722-aeb8-981b415cc9d6 None None] MFA Rules not processed for user `14a6ec692df645c7878073d04fd97dd8`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:58.224176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-06dc86d4-69d7-4722-aeb8-981b415cc9d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:58.229826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0e92045d-d756-41a0-9d01-1a3f09d552ad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9bc52069875c413195933c9143cc6ff6', 'request_id': 'req-0e92045d-d756-41a0-9d01-1a3f09d552ad'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:58.249769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0e92045d-d756-41a0-9d01-1a3f09d552ad admin admin] The token cache is being invalidate because project 9bc52069875c413195933c9143cc6ff6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:58.250492 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0e92045d-d756-41a0-9d01-1a3f09d552ad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:58.251481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0e92045d-d756-41a0-9d01-1a3f09d552ad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:58.253396 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1056/2126] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:58 2022] DELETE /identity/v3/projects/9bc52069875c413195933c9143cc6ff6 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:58.257854 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e92045d-d756-41a0-9d01-1a3f09d552ad admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:58.260614 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0e92045d-d756-41a0-9d01-1a3f09d552ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:58.385707 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1071/2127] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:58 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:19:58.390586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06dc86d4-69d7-4722-aeb8-981b415cc9d6 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:58.393442 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-06dc86d4-69d7-4722-aeb8-981b415cc9d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:58.403813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e92045d-d756-41a0-9d01-1a3f09d552ad admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:58.422807 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742eb167-b970-46a3-97e8-de4f73ee7057 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:58.423561 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742eb167-b970-46a3-97e8-de4f73ee7057 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:58.423838 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742eb167-b970-46a3-97e8-de4f73ee7057 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:58.424108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742eb167-b970-46a3-97e8-de4f73ee7057 admin admin] PATH_INFO: `/v3/users/4d964cf647a04116aac7b0776a888b38` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:58.425745 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742eb167-b970-46a3-97e8-de4f73ee7057 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4d964cf647a04116aac7b0776a888b38)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:58.427729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742eb167-b970-46a3-97e8-de4f73ee7057 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:58.433156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06dc86d4-69d7-4722-aeb8-981b415cc9d6 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:58.449895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4185c795-98ba-4696-8f55-15bb3d6eb095 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:58.450890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4185c795-98ba-4696-8f55-15bb3d6eb095 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:58.451823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4185c795-98ba-4696-8f55-15bb3d6eb095 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:58.452301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4185c795-98ba-4696-8f55-15bb3d6eb095 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:58.452752 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4185c795-98ba-4696-8f55-15bb3d6eb095 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:58.454870 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4185c795-98ba-4696-8f55-15bb3d6eb095 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:58.523590 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-742eb167-b970-46a3-97e8-de4f73ee7057 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4d964cf647a04116aac7b0776a888b38'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:58.524777 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-742eb167-b970-46a3-97e8-de4f73ee7057 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4d964cf647a04116aac7b0776a888b38'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:58.541554 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1072/2128] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:58 2022] POST /identity/v3/projects => generated 367 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:58.546045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4185c795-98ba-4696-8f55-15bb3d6eb095 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:58.547906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-742eb167-b970-46a3-97e8-de4f73ee7057 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4d964cf647a04116aac7b0776a888b38'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:58.548739 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-742eb167-b970-46a3-97e8-de4f73ee7057 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4d964cf647a04116aac7b0776a888b38'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:58.573739 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-742eb167-b970-46a3-97e8-de4f73ee7057 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4d964cf647a04116aac7b0776a888b38'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:58.628320 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1057/2129] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:19:58 2022] DELETE /identity/v3/users/4d964cf647a04116aac7b0776a888b38 => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:58.718398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4185c795-98ba-4696-8f55-15bb3d6eb095 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:58.737652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:58.855901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5a3590-c982-4442-875d-1c0c757d0470 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:58.859645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5a3590-c982-4442-875d-1c0c757d0470 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:58.860344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5a3590-c982-4442-875d-1c0c757d0470 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:58.860825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5a3590-c982-4442-875d-1c0c757d0470 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:58.862051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5a3590-c982-4442-875d-1c0c757d0470 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:58.865058 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5a3590-c982-4442-875d-1c0c757d0470 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:58.931311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1073/2130] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:19:58 2022] POST /identity/v3/users => generated 290 bytes in 383 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:19:58.932706 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742eb167-b970-46a3-97e8-de4f73ee7057 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:58.934881 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-742eb167-b970-46a3-97e8-de4f73ee7057 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:58.988561 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742eb167-b970-46a3-97e8-de4f73ee7057 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:59.005394 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6c0844-bbbd-4fd5-a2d4-914ad0bfef67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:59.006232 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6c0844-bbbd-4fd5-a2d4-914ad0bfef67 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:59.006735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6c0844-bbbd-4fd5-a2d4-914ad0bfef67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:59.008729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6c0844-bbbd-4fd5-a2d4-914ad0bfef67 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:59.009699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6c0844-bbbd-4fd5-a2d4-914ad0bfef67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:59.012039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6c0844-bbbd-4fd5-a2d4-914ad0bfef67 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:59.030588 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1058/2131] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:58 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:59.034636 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5a3590-c982-4442-875d-1c0c757d0470 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:59.046640 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5a3590-c982-4442-875d-1c0c757d0470 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:59.051833 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd280a1-a135-4c67-abb8-18957f539817 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:59.052500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd280a1-a135-4c67-abb8-18957f539817 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:59.052842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd280a1-a135-4c67-abb8-18957f539817 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:59.053056 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd280a1-a135-4c67-abb8-18957f539817 admin admin] PATH_INFO: `/v3/projects/105a2bde9908451b81c509e7f45913a3/users/d8b28e817ef84fecb0d42aaad09c3004/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:59.070883 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd280a1-a135-4c67-abb8-18957f539817 admin admin] RBAC: Authorizing `identity:create_grant(project_id=105a2bde9908451b81c509e7f45913a3, user_id=d8b28e817ef84fecb0d42aaad09c3004, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:59.073516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd280a1-a135-4c67-abb8-18957f539817 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:59.088867 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1074/2132] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:59 2022] PUT /identity/v3/projects/105a2bde9908451b81c509e7f45913a3/users/d8b28e817ef84fecb0d42aaad09c3004/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:59.092920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6c0844-bbbd-4fd5-a2d4-914ad0bfef67 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:59.098628 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:59.104990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6c0844-bbbd-4fd5-a2d4-914ad0bfef67 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:59.164506 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd280a1-a135-4c67-abb8-18957f539817 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:59.175314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3dd280a1-a135-4c67-abb8-18957f539817 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:59.254899 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd280a1-a135-4c67-abb8-18957f539817 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:59.259556 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6d5962-aeac-47fb-be80-fa99ede72529 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:59.260333 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6d5962-aeac-47fb-be80-fa99ede72529 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:59.260713 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6d5962-aeac-47fb-be80-fa99ede72529 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:59.261085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6d5962-aeac-47fb-be80-fa99ede72529 admin admin] PATH_INFO: `/v3/projects/d814167c18944a57bfeff3002b244347` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:59.262344 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6d5962-aeac-47fb-be80-fa99ede72529 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d814167c18944a57bfeff3002b244347)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:59.264630 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6d5962-aeac-47fb-be80-fa99ede72529 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:59.268818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015da411-0e69-4245-92ec-0f885bcc94ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:59.269683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015da411-0e69-4245-92ec-0f885bcc94ee admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:59.270214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015da411-0e69-4245-92ec-0f885bcc94ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:59.270629 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015da411-0e69-4245-92ec-0f885bcc94ee admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:59.271768 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015da411-0e69-4245-92ec-0f885bcc94ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:59.274452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015da411-0e69-4245-92ec-0f885bcc94ee admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:59.298775 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1059/2133] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:19:59 2022] GET /identity/v3/roles => generated 1420 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:19:59.302330 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015da411-0e69-4245-92ec-0f885bcc94ee admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:59.321424 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015da411-0e69-4245-92ec-0f885bcc94ee admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:59.331477 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-da6d5962-aeac-47fb-be80-fa99ede72529 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd814167c18944a57bfeff3002b244347', 'request_id': 'req-da6d5962-aeac-47fb-be80-fa99ede72529'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:59.339973 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-da6d5962-aeac-47fb-be80-fa99ede72529 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd814167c18944a57bfeff3002b244347', 'request_id': 'req-da6d5962-aeac-47fb-be80-fa99ede72529'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:59.366946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-da6d5962-aeac-47fb-be80-fa99ede72529 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd814167c18944a57bfeff3002b244347', 'request_id': 'req-da6d5962-aeac-47fb-be80-fa99ede72529'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:59.393498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-da6d5962-aeac-47fb-be80-fa99ede72529 admin admin] The token cache is being invalidate because project d814167c18944a57bfeff3002b244347 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:19:59.394000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-da6d5962-aeac-47fb-be80-fa99ede72529 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:59.400293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-da6d5962-aeac-47fb-be80-fa99ede72529 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:19:59.411820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1075/2134] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:19:59 2022] DELETE /identity/v3/projects/d814167c18944a57bfeff3002b244347 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:19:59.411820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0afb7e-a971-4e8e-863b-0bf2b52125e6 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:59.411820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0afb7e-a971-4e8e-863b-0bf2b52125e6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:59.411820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0afb7e-a971-4e8e-863b-0bf2b52125e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:59.435474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:59.442780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4c0afb7e-a971-4e8e-863b-0bf2b52125e6 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:59.452186 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4c0afb7e-a971-4e8e-863b-0bf2b52125e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:59.456458 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2415d9-aae7-4286-9698-a5ceddb299a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:59.458054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2415d9-aae7-4286-9698-a5ceddb299a8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:59.458054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2415d9-aae7-4286-9698-a5ceddb299a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:59.458054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2415d9-aae7-4286-9698-a5ceddb299a8 admin admin] PATH_INFO: `/v3/projects/105a2bde9908451b81c509e7f45913a3/users/d8b28e817ef84fecb0d42aaad09c3004/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:59.462394 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2415d9-aae7-4286-9698-a5ceddb299a8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=105a2bde9908451b81c509e7f45913a3, user_id=d8b28e817ef84fecb0d42aaad09c3004, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:59.468168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2415d9-aae7-4286-9698-a5ceddb299a8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:59.475025 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1076/2135] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:59.483926 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695a54a5-2084-4a9e-a1b0-aefdf0da25a8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:59.484278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695a54a5-2084-4a9e-a1b0-aefdf0da25a8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:59.484654 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695a54a5-2084-4a9e-a1b0-aefdf0da25a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:59.501710 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1060/2136] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:19:59 2022] PUT /identity/v3/projects/105a2bde9908451b81c509e7f45913a3/users/d8b28e817ef84fecb0d42aaad09c3004/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:19:59.503529 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2f9e32-91cf-4452-ab36-a129c3d98d06 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:59.504091 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2f9e32-91cf-4452-ab36-a129c3d98d06 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:59.504507 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2f9e32-91cf-4452-ab36-a129c3d98d06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:59.532307 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-695a54a5-2084-4a9e-a1b0-aefdf0da25a8 None None] MFA Rules not processed for user `ee8f5c0bb6c947dfbd35022ecfac08c4`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:59.534033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-695a54a5-2084-4a9e-a1b0-aefdf0da25a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:59.557369 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ff2f9e32-91cf-4452-ab36-a129c3d98d06 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:59.558745 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ff2f9e32-91cf-4452-ab36-a129c3d98d06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:59.648273 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1077/2137] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:59 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:59.657474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95a9d08-4690-4b67-92b3-9e29e48c0f14 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:59.657770 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95a9d08-4690-4b67-92b3-9e29e48c0f14 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:59.658050 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95a9d08-4690-4b67-92b3-9e29e48c0f14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:59.675487 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1061/2138] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:59.677238 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2f9e32-91cf-4452-ab36-a129c3d98d06 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:59.678745 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ff2f9e32-91cf-4452-ab36-a129c3d98d06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:59.691886 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a95a9d08-4690-4b67-92b3-9e29e48c0f14 None None] MFA Rules not processed for user `d8b28e817ef84fecb0d42aaad09c3004`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:19:59.693414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a95a9d08-4690-4b67-92b3-9e29e48c0f14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:59.725926 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:59.777214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2f9e32-91cf-4452-ab36-a129c3d98d06 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:59.783727 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0585cc-3cf6-4fcd-8815-6650eec15065 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:59.784175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1078/2139] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:19:59 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:19:59.784904 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0585cc-3cf6-4fcd-8815-6650eec15065 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:59.785347 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95a9d08-4690-4b67-92b3-9e29e48c0f14 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:59.785703 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0585cc-3cf6-4fcd-8815-6650eec15065 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:59.786373 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0585cc-3cf6-4fcd-8815-6650eec15065 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:59.793126 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6c0585cc-3cf6-4fcd-8815-6650eec15065 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:59.802669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95a9d08-4690-4b67-92b3-9e29e48c0f14 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:59.807971 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8c6d99-0c37-45bb-a706-5ff1f52b6d14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:59.809563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8c6d99-0c37-45bb-a706-5ff1f52b6d14 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:59.809907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8c6d99-0c37-45bb-a706-5ff1f52b6d14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:59.810243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8c6d99-0c37-45bb-a706-5ff1f52b6d14 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:59.811371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8c6d99-0c37-45bb-a706-5ff1f52b6d14 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:19:59.811739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8c6d99-0c37-45bb-a706-5ff1f52b6d14 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:59.813462 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8c6d99-0c37-45bb-a706-5ff1f52b6d14 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:59.832448 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1079/2140] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:19:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:19:59.832448 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8c6d99-0c37-45bb-a706-5ff1f52b6d14 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:59.833735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0c8c6d99-0c37-45bb-a706-5ff1f52b6d14 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:19:59.912420 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8c6d99-0c37-45bb-a706-5ff1f52b6d14 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:59.916454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef47dd0-75e7-4adb-87c3-fbaf8af2e08e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:59.917012 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef47dd0-75e7-4adb-87c3-fbaf8af2e08e service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:59.917287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef47dd0-75e7-4adb-87c3-fbaf8af2e08e service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:59.917616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef47dd0-75e7-4adb-87c3-fbaf8af2e08e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:59.928629 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef47dd0-75e7-4adb-87c3-fbaf8af2e08e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:59.929214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0585cc-3cf6-4fcd-8815-6650eec15065 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:59.929563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef47dd0-75e7-4adb-87c3-fbaf8af2e08e service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:59.930830 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0585cc-3cf6-4fcd-8815-6650eec15065 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:19:59.938724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:19:59.947659 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1080/2141] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:59 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:59.948384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef47dd0-75e7-4adb-87c3-fbaf8af2e08e service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:19:59.951504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1062/2142] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:19:59 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 275 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:19:59.963580 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef47dd0-75e7-4adb-87c3-fbaf8af2e08e service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:19:59.973126 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e988096b-bf24-4c8b-8702-81e4d49b3ece admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:19:59.973827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e988096b-bf24-4c8b-8702-81e4d49b3ece admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:19:59.974108 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e988096b-bf24-4c8b-8702-81e4d49b3ece admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:19:59.974489 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e988096b-bf24-4c8b-8702-81e4d49b3ece admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:19:59.975314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e988096b-bf24-4c8b-8702-81e4d49b3ece admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:19:59.977945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e988096b-bf24-4c8b-8702-81e4d49b3ece admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:00.021941 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1081/2143] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:19:59 2022] POST /identity/v3/projects => generated 383 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:20:00.025816 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0585cc-3cf6-4fcd-8815-6650eec15065 service nova] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:00.040519 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0585cc-3cf6-4fcd-8815-6650eec15065 service nova] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:00.122999 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e988096b-bf24-4c8b-8702-81e4d49b3ece admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:00.170770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca6d2a0-cf80-45be-9a63-78d6ade7b25c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:00.172610 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca6d2a0-cf80-45be-9a63-78d6ade7b25c admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:00.175776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca6d2a0-cf80-45be-9a63-78d6ade7b25c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:00.176285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca6d2a0-cf80-45be-9a63-78d6ade7b25c admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:00.177270 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca6d2a0-cf80-45be-9a63-78d6ade7b25c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:00.179981 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca6d2a0-cf80-45be-9a63-78d6ade7b25c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:00.238807 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1063/2144] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:20:00 2022] POST /identity/v3/users => generated 292 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:20:00.245118 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca6d2a0-cf80-45be-9a63-78d6ade7b25c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:00.270824 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca6d2a0-cf80-45be-9a63-78d6ade7b25c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:00.286136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b159b8-fd98-4fd8-8359-70a821bf6557 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:00.288262 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b159b8-fd98-4fd8-8359-70a821bf6557 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:00.288846 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b159b8-fd98-4fd8-8359-70a821bf6557 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:00.289217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b159b8-fd98-4fd8-8359-70a821bf6557 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:00.291180 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b159b8-fd98-4fd8-8359-70a821bf6557 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:00.295848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b159b8-fd98-4fd8-8359-70a821bf6557 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:00.309351 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e988096b-bf24-4c8b-8702-81e4d49b3ece admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:00.315630 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1064/2145] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:20:00 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:20:00.316918 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b159b8-fd98-4fd8-8359-70a821bf6557 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:00.323332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d07a13-c3f2-4f79-9c68-16c8733f8fc9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:00.324046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d07a13-c3f2-4f79-9c68-16c8733f8fc9 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:00.324318 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d07a13-c3f2-4f79-9c68-16c8733f8fc9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:00.324609 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d07a13-c3f2-4f79-9c68-16c8733f8fc9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:00.327057 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e5d07a13-c3f2-4f79-9c68-16c8733f8fc9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:00.342807 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b159b8-fd98-4fd8-8359-70a821bf6557 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:00.349203 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db580fb-c9d5-486d-aa54-98b1367ca30b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:00.350118 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db580fb-c9d5-486d-aa54-98b1367ca30b admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:00.350571 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db580fb-c9d5-486d-aa54-98b1367ca30b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:00.351232 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db580fb-c9d5-486d-aa54-98b1367ca30b admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:00.352101 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db580fb-c9d5-486d-aa54-98b1367ca30b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:00.353961 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db580fb-c9d5-486d-aa54-98b1367ca30b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:00.372818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1065/2146] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:20:00 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:20:00.373946 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db580fb-c9d5-486d-aa54-98b1367ca30b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:00.385573 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db580fb-c9d5-486d-aa54-98b1367ca30b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:00.393368 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1cb3ce-b449-4a74-a048-af7f24277c9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:00.394322 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1cb3ce-b449-4a74-a048-af7f24277c9c admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:00.394749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1cb3ce-b449-4a74-a048-af7f24277c9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:00.395262 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1cb3ce-b449-4a74-a048-af7f24277c9c admin admin] PATH_INFO: `/v3/projects/41877dc874614e8fa303b2a64948f7b6/users/3772a22140174161adc0f6e8dcb701b1/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:00.434213 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d07a13-c3f2-4f79-9c68-16c8733f8fc9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:00.440218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d07a13-c3f2-4f79-9c68-16c8733f8fc9 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:00.452794 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1cb3ce-b449-4a74-a048-af7f24277c9c admin admin] RBAC: Authorizing `identity:create_grant(project_id=41877dc874614e8fa303b2a64948f7b6, user_id=3772a22140174161adc0f6e8dcb701b1, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:00.462098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1cb3ce-b449-4a74-a048-af7f24277c9c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:00.478665 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1082/2147] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:20:00 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:20:00.530450 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1066/2148] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:20:00 2022] PUT /identity/v3/projects/41877dc874614e8fa303b2a64948f7b6/users/3772a22140174161adc0f6e8dcb701b1/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:20:00.531836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4325e882-cf68-4d01-8055-9c3b6a7c5480 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:00.532163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4325e882-cf68-4d01-8055-9c3b6a7c5480 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:00.532466 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4325e882-cf68-4d01-8055-9c3b6a7c5480 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:00.600062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4325e882-cf68-4d01-8055-9c3b6a7c5480 None None] MFA Rules not processed for user `3772a22140174161adc0f6e8dcb701b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:00.602552 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4325e882-cf68-4d01-8055-9c3b6a7c5480 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:00.809359 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1083/2149] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:00 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:00.826577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1cb3ce-b449-4a74-a048-af7f24277c9c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:00.826577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cf1cb3ce-b449-4a74-a048-af7f24277c9c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:00.853706 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:00.982665 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1cb3ce-b449-4a74-a048-af7f24277c9c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:00.993392 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1fc929-44fc-4e12-b68b-39e7f2f448d0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:00.994097 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1fc929-44fc-4e12-b68b-39e7f2f448d0 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:00.994378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1fc929-44fc-4e12-b68b-39e7f2f448d0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:00.994652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1fc929-44fc-4e12-b68b-39e7f2f448d0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:01.017594 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1fc929-44fc-4e12-b68b-39e7f2f448d0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:01.019101 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1fc929-44fc-4e12-b68b-39e7f2f448d0 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:01.037538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1067/2150] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:20:00 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:20:03.250174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9000ff86-d831-47ba-8c75-ea59438ff049 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:03.251573 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9000ff86-d831-47ba-8c75-ea59438ff049 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:03.251967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9000ff86-d831-47ba-8c75-ea59438ff049 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:03.253562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1084/2151] 10.222.0.15 () {54 vars in 921 bytes} [Tue Sep 13 01:20:03 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:20:03.272189 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e981388d-ecb4-4d1b-83fc-a2ec6b2bbca3 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:03.272189 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e981388d-ecb4-4d1b-83fc-a2ec6b2bbca3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:03.272633 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e981388d-ecb4-4d1b-83fc-a2ec6b2bbca3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:03.301285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e981388d-ecb4-4d1b-83fc-a2ec6b2bbca3 None None] MFA Rules not processed for user `8eaff1b843a145a49994dcf089f44315`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:03.303065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e981388d-ecb4-4d1b-83fc-a2ec6b2bbca3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:03.353514 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9000ff86-d831-47ba-8c75-ea59438ff049 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:03.366550 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9000ff86-d831-47ba-8c75-ea59438ff049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:03.416480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:03.422904 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1068/2152] 10.222.0.15 () {58 vars in 1018 bytes} [Tue Sep 13 01:20:03 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:03.436946 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e12f599-a2e2-4183-beb0-8584f231979a None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:03.437737 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e12f599-a2e2-4183-beb0-8584f231979a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:03.438282 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e12f599-a2e2-4183-beb0-8584f231979a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:03.461416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8e12f599-a2e2-4183-beb0-8584f231979a None None] MFA Rules not processed for user `85178dc2280c4a54970fc207d1746608`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:03.462797 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8e12f599-a2e2-4183-beb0-8584f231979a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:03.502309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9000ff86-d831-47ba-8c75-ea59438ff049 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:03.511188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41c0a18-b877-42c1-b63f-10c1cb67fa26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:03.511843 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41c0a18-b877-42c1-b63f-10c1cb67fa26 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:03.512180 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41c0a18-b877-42c1-b63f-10c1cb67fa26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:03.512450 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41c0a18-b877-42c1-b63f-10c1cb67fa26 admin admin] PATH_INFO: `/v3/users/06797d4f56954444b3317f2339f020c4` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:03.513848 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41c0a18-b877-42c1-b63f-10c1cb67fa26 admin admin] RBAC: Authorizing `identity:delete_user(user_id=06797d4f56954444b3317f2339f020c4)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:03.516106 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41c0a18-b877-42c1-b63f-10c1cb67fa26 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:03.525653 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1069/2153] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:03 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:03.526577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e12f599-a2e2-4183-beb0-8584f231979a None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:03.531589 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8e12f599-a2e2-4183-beb0-8584f231979a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:03.551003 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e12f599-a2e2-4183-beb0-8584f231979a None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:03.565453 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcba5bb7-f4be-4fba-802d-fadbeccd9ae9 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8eaff1b843a145a49994dcf089f44315', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:03.566074 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcba5bb7-f4be-4fba-802d-fadbeccd9ae9 service placement] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:03.566353 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcba5bb7-f4be-4fba-802d-fadbeccd9ae9 service placement] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:03.566613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcba5bb7-f4be-4fba-802d-fadbeccd9ae9 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:03.576846 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcba5bb7-f4be-4fba-802d-fadbeccd9ae9 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:03.578498 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcba5bb7-f4be-4fba-802d-fadbeccd9ae9 service placement] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:03.580055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f41c0a18-b877-42c1-b63f-10c1cb67fa26 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '06797d4f56954444b3317f2339f020c4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:03.581052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f41c0a18-b877-42c1-b63f-10c1cb67fa26 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '06797d4f56954444b3317f2339f020c4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:03.590501 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1070/2154] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:20:03 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:20:03.596623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f41c0a18-b877-42c1-b63f-10c1cb67fa26 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '06797d4f56954444b3317f2339f020c4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:03.597399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f41c0a18-b877-42c1-b63f-10c1cb67fa26 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '06797d4f56954444b3317f2339f020c4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:03.618562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f41c0a18-b877-42c1-b63f-10c1cb67fa26 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '06797d4f56954444b3317f2339f020c4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:03.632846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1085/2155] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:20:03 2022] DELETE /identity/v3/users/06797d4f56954444b3317f2339f020c4 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:20:03.871645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcba5bb7-f4be-4fba-802d-fadbeccd9ae9 service placement] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:03.893341 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcba5bb7-f4be-4fba-802d-fadbeccd9ae9 service placement] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:03.967384 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:03.992936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6699ec7-91e0-43e5-a31d-0f9f10393f57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:03.993520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6699ec7-91e0-43e5-a31d-0f9f10393f57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:03.993865 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6699ec7-91e0-43e5-a31d-0f9f10393f57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:03.994068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6699ec7-91e0-43e5-a31d-0f9f10393f57 admin admin] PATH_INFO: `/v3/projects/6475b1b1e54c4b0eb86021d82021de78` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:03.995479 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6699ec7-91e0-43e5-a31d-0f9f10393f57 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6475b1b1e54c4b0eb86021d82021de78)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:03.999402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6699ec7-91e0-43e5-a31d-0f9f10393f57 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:04.052937 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a0a6fc-f297-4e3a-9dc6-ebb57d2f1364 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:04.053244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a0a6fc-f297-4e3a-9dc6-ebb57d2f1364 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:04.053484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a0a6fc-f297-4e3a-9dc6-ebb57d2f1364 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:04.078892 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-80a0a6fc-f297-4e3a-9dc6-ebb57d2f1364 None None] MFA Rules not processed for user `e28a0ba45389417ba8ae92376dba87ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:04.079142 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-80a0a6fc-f297-4e3a-9dc6-ebb57d2f1364 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:04.094349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e6699ec7-91e0-43e5-a31d-0f9f10393f57 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6475b1b1e54c4b0eb86021d82021de78', 'request_id': 'req-e6699ec7-91e0-43e5-a31d-0f9f10393f57'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:04.103824 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e6699ec7-91e0-43e5-a31d-0f9f10393f57 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6475b1b1e54c4b0eb86021d82021de78', 'request_id': 'req-e6699ec7-91e0-43e5-a31d-0f9f10393f57'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:04.116834 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e6699ec7-91e0-43e5-a31d-0f9f10393f57 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6475b1b1e54c4b0eb86021d82021de78', 'request_id': 'req-e6699ec7-91e0-43e5-a31d-0f9f10393f57'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:04.144070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e6699ec7-91e0-43e5-a31d-0f9f10393f57 admin admin] The token cache is being invalidate because project 6475b1b1e54c4b0eb86021d82021de78 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:20:04.144704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e6699ec7-91e0-43e5-a31d-0f9f10393f57 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:04.145660 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e6699ec7-91e0-43e5-a31d-0f9f10393f57 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:04.150210 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1071/2156] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:20:03 2022] DELETE /identity/v3/projects/6475b1b1e54c4b0eb86021d82021de78 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:20:04.155593 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6699ec7-91e0-43e5-a31d-0f9f10393f57 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:04.157187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e6699ec7-91e0-43e5-a31d-0f9f10393f57 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:04.195426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1086/2157] 10.222.0.15 () {58 vars in 1009 bytes} [Tue Sep 13 01:20:04 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:20:04.196934 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2869f3c8-06ad-4f8e-8f10-7cff476488d5 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:04.197451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2869f3c8-06ad-4f8e-8f10-7cff476488d5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:04.198013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2869f3c8-06ad-4f8e-8f10-7cff476488d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:04.252316 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-2869f3c8-06ad-4f8e-8f10-7cff476488d5 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:04.254198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2869f3c8-06ad-4f8e-8f10-7cff476488d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:04.313252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6699ec7-91e0-43e5-a31d-0f9f10393f57 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:04.319767 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be32a3f-dcc3-4690-9751-1799408697c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:04.320814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be32a3f-dcc3-4690-9751-1799408697c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:04.321410 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be32a3f-dcc3-4690-9751-1799408697c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:04.321884 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be32a3f-dcc3-4690-9751-1799408697c0 admin admin] PATH_INFO: `/v3/users/3772a22140174161adc0f6e8dcb701b1` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:04.323317 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be32a3f-dcc3-4690-9751-1799408697c0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3772a22140174161adc0f6e8dcb701b1)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:04.326221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be32a3f-dcc3-4690-9751-1799408697c0 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:04.330969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:04.364160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1087/2158] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:04.368074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2869f3c8-06ad-4f8e-8f10-7cff476488d5 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:04.373124 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2869f3c8-06ad-4f8e-8f10-7cff476488d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:04.414820 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4be32a3f-dcc3-4690-9751-1799408697c0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3772a22140174161adc0f6e8dcb701b1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:04.415562 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4be32a3f-dcc3-4690-9751-1799408697c0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3772a22140174161adc0f6e8dcb701b1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:04.425722 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4be32a3f-dcc3-4690-9751-1799408697c0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3772a22140174161adc0f6e8dcb701b1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:04.427412 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4be32a3f-dcc3-4690-9751-1799408697c0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3772a22140174161adc0f6e8dcb701b1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:04.435504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4be32a3f-dcc3-4690-9751-1799408697c0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3772a22140174161adc0f6e8dcb701b1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:04.447562 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:04.461982 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1072/2159] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:20:04 2022] DELETE /identity/v3/users/3772a22140174161adc0f6e8dcb701b1 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:20:04.463265 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e14d881-6be1-405b-b10a-1460529658fd None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:04.463564 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e14d881-6be1-405b-b10a-1460529658fd None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:04.463835 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e14d881-6be1-405b-b10a-1460529658fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:04.487414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2869f3c8-06ad-4f8e-8f10-7cff476488d5 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:04.490450 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6e14d881-6be1-405b-b10a-1460529658fd None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:04.491680 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6e14d881-6be1-405b-b10a-1460529658fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:04.629736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1073/2160] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:04.631340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52f128f-7e0c-4e88-aa0f-7ba7401b6482 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:04.631681 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52f128f-7e0c-4e88-aa0f-7ba7401b6482 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:04.632025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52f128f-7e0c-4e88-aa0f-7ba7401b6482 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:04.637660 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c233e2-3027-4568-9f74-37d7b8294957 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d7d8ed4ede4ddca192fce0a497689d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:04.638286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c233e2-3027-4568-9f74-37d7b8294957 service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:04.638564 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c233e2-3027-4568-9f74-37d7b8294957 service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:04.638829 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c233e2-3027-4568-9f74-37d7b8294957 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:04.649090 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b7c233e2-3027-4568-9f74-37d7b8294957 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:04.655962 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-a52f128f-7e0c-4e88-aa0f-7ba7401b6482 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:04.657229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a52f128f-7e0c-4e88-aa0f-7ba7401b6482 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:04.668023 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1074/2161] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:04.669185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52f128f-7e0c-4e88-aa0f-7ba7401b6482 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:04.684444 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52f128f-7e0c-4e88-aa0f-7ba7401b6482 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:04.691470 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7784885d-90f4-4d00-a85b-7914159817a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:04.692367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7784885d-90f4-4d00-a85b-7914159817a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:04.692714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7784885d-90f4-4d00-a85b-7914159817a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:04.693093 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7784885d-90f4-4d00-a85b-7914159817a3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:04.693815 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7784885d-90f4-4d00-a85b-7914159817a3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:20:04.694187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7784885d-90f4-4d00-a85b-7914159817a3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:04.696005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7784885d-90f4-4d00-a85b-7914159817a3 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:04.713411 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1075/2162] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:20:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:20:04.714601 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7784885d-90f4-4d00-a85b-7914159817a3 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:04.720035 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7784885d-90f4-4d00-a85b-7914159817a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:04.775231 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c233e2-3027-4568-9f74-37d7b8294957 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:04.778201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7c233e2-3027-4568-9f74-37d7b8294957 service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:04.799640 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1088/2163] 10.222.0.15 () {60 vars in 1384 bytes} [Tue Sep 13 01:20:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 01:20:04.800481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c233e2-3027-4568-9f74-37d7b8294957 service swift] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:04.815446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c233e2-3027-4568-9f74-37d7b8294957 service swift] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:04.823833 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb49d759-4345-441e-a0fe-702599786749 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:04.824518 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb49d759-4345-441e-a0fe-702599786749 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:04.825409 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb49d759-4345-441e-a0fe-702599786749 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:04.825663 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb49d759-4345-441e-a0fe-702599786749 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:04.826333 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb49d759-4345-441e-a0fe-702599786749 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:04.828308 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb49d759-4345-441e-a0fe-702599786749 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:04.837725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7784885d-90f4-4d00-a85b-7914159817a3 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:04.844553 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037d4074-4514-4876-aa0b-b29be2d5ad66 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:04.845230 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037d4074-4514-4876-aa0b-b29be2d5ad66 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:04.845496 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037d4074-4514-4876-aa0b-b29be2d5ad66 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:04.845773 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037d4074-4514-4876-aa0b-b29be2d5ad66 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:04.866686 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037d4074-4514-4876-aa0b-b29be2d5ad66 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:04.868196 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037d4074-4514-4876-aa0b-b29be2d5ad66 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:04.878340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1076/2164] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:20:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:20:04.888670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1089/2165] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:20:04 2022] POST /identity/v3/projects => generated 383 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:20:04.892841 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037d4074-4514-4876-aa0b-b29be2d5ad66 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:04.912603 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037d4074-4514-4876-aa0b-b29be2d5ad66 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:05.011373 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5013b9f9-7d19-4047-afaf-62e1acf86267 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:05.013112 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5013b9f9-7d19-4047-afaf-62e1acf86267 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:05.014563 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5013b9f9-7d19-4047-afaf-62e1acf86267 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:05.015150 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5013b9f9-7d19-4047-afaf-62e1acf86267 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:05.016347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5013b9f9-7d19-4047-afaf-62e1acf86267 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:05.018518 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5013b9f9-7d19-4047-afaf-62e1acf86267 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:05.051326 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1077/2166] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:20:04 2022] POST /identity/v3/users => generated 292 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:20:05.057647 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb49d759-4345-441e-a0fe-702599786749 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:05.073658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb49d759-4345-441e-a0fe-702599786749 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:05.082410 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2986f5-7e5a-4f1c-ac16-02ea40889225 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:05.084111 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2986f5-7e5a-4f1c-ac16-02ea40889225 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:05.084911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2986f5-7e5a-4f1c-ac16-02ea40889225 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:05.085384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2986f5-7e5a-4f1c-ac16-02ea40889225 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:05.095000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2986f5-7e5a-4f1c-ac16-02ea40889225 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:05.097188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2986f5-7e5a-4f1c-ac16-02ea40889225 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:05.100001 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:05.119587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5013b9f9-7d19-4047-afaf-62e1acf86267 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:05.124641 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1090/2167] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:20:05 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:20:05.137713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5013b9f9-7d19-4047-afaf-62e1acf86267 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:05.147476 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb574f6-eb0b-462e-8616-ce4412dc6a0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:05.148413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb574f6-eb0b-462e-8616-ce4412dc6a0a admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:05.150539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb574f6-eb0b-462e-8616-ce4412dc6a0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:05.150834 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb574f6-eb0b-462e-8616-ce4412dc6a0a admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:05.151689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb574f6-eb0b-462e-8616-ce4412dc6a0a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:05.153741 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb574f6-eb0b-462e-8616-ce4412dc6a0a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:05.155201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:DELETE=1,keystone:SELECT=96,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:05.171615 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1078/2168] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:20:05 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:20:05.176877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2986f5-7e5a-4f1c-ac16-02ea40889225 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:05.192808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2986f5-7e5a-4f1c-ac16-02ea40889225 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:05.198727 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474065ef-9cf2-4790-87a6-b1fbe92d7909 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:05.200155 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474065ef-9cf2-4790-87a6-b1fbe92d7909 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:05.200513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474065ef-9cf2-4790-87a6-b1fbe92d7909 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:05.200843 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474065ef-9cf2-4790-87a6-b1fbe92d7909 admin admin] PATH_INFO: `/v3/projects/30ce8239a9b34af685743812ce53ef19/users/776733af5b6a4115b2a11d67666a379d/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:05.222616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474065ef-9cf2-4790-87a6-b1fbe92d7909 admin admin] RBAC: Authorizing `identity:create_grant(project_id=30ce8239a9b34af685743812ce53ef19, user_id=776733af5b6a4115b2a11d67666a379d, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:05.226474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474065ef-9cf2-4790-87a6-b1fbe92d7909 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:05.250887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1091/2169] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:20:05 2022] PUT /identity/v3/projects/30ce8239a9b34af685743812ce53ef19/users/776733af5b6a4115b2a11d67666a379d/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:20:05.258916 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ea5fbf-1ec6-495e-a8aa-e044da69a9b3 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:05.258916 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ea5fbf-1ec6-495e-a8aa-e044da69a9b3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:05.258916 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ea5fbf-1ec6-495e-a8aa-e044da69a9b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:05.282265 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474065ef-9cf2-4790-87a6-b1fbe92d7909 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:05.285177 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-474065ef-9cf2-4790-87a6-b1fbe92d7909 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:05.287194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-87ea5fbf-1ec6-495e-a8aa-e044da69a9b3 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:05.289182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-87ea5fbf-1ec6-495e-a8aa-e044da69a9b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:05.397151 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1079/2170] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:05.414744 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ea5fbf-1ec6-495e-a8aa-e044da69a9b3 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:05.420153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-87ea5fbf-1ec6-495e-a8aa-e044da69a9b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:05.452994 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474065ef-9cf2-4790-87a6-b1fbe92d7909 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:05.459816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc0fa94-2929-41fd-8fcc-bc978703917a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:05.461604 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc0fa94-2929-41fd-8fcc-bc978703917a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:05.462023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc0fa94-2929-41fd-8fcc-bc978703917a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:05.462613 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc0fa94-2929-41fd-8fcc-bc978703917a admin admin] PATH_INFO: `/v3/projects/41877dc874614e8fa303b2a64948f7b6` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:05.465892 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc0fa94-2929-41fd-8fcc-bc978703917a admin admin] RBAC: Authorizing `identity:delete_project(project_id=41877dc874614e8fa303b2a64948f7b6)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:05.468733 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc0fa94-2929-41fd-8fcc-bc978703917a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:05.546891 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9fc0fa94-2929-41fd-8fcc-bc978703917a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '41877dc874614e8fa303b2a64948f7b6', 'request_id': 'req-9fc0fa94-2929-41fd-8fcc-bc978703917a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:05.552731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ea5fbf-1ec6-495e-a8aa-e044da69a9b3 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:05.552872 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9fc0fa94-2929-41fd-8fcc-bc978703917a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '41877dc874614e8fa303b2a64948f7b6', 'request_id': 'req-9fc0fa94-2929-41fd-8fcc-bc978703917a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:05.567396 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9fc0fa94-2929-41fd-8fcc-bc978703917a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '41877dc874614e8fa303b2a64948f7b6', 'request_id': 'req-9fc0fa94-2929-41fd-8fcc-bc978703917a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:05.606546 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9fc0fa94-2929-41fd-8fcc-bc978703917a admin admin] The token cache is being invalidate because project 41877dc874614e8fa303b2a64948f7b6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:20:05.606860 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9fc0fa94-2929-41fd-8fcc-bc978703917a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:05.607880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-9fc0fa94-2929-41fd-8fcc-bc978703917a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:05.613861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1092/2171] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:20:05 2022] DELETE /identity/v3/projects/41877dc874614e8fa303b2a64948f7b6 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:20:05.636537 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f98a316-2ad2-480f-ad49-22c9e407393b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:05.637366 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f98a316-2ad2-480f-ad49-22c9e407393b service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:05.637790 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f98a316-2ad2-480f-ad49-22c9e407393b service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:05.638370 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f98a316-2ad2-480f-ad49-22c9e407393b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:05.648317 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2f98a316-2ad2-480f-ad49-22c9e407393b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:05.758660 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f98a316-2ad2-480f-ad49-22c9e407393b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:05.760470 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f98a316-2ad2-480f-ad49-22c9e407393b service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:05.772511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1080/2172] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:20:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:20:05.790037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5602bce0-90ee-438d-ae39-d3d5a2ef1c63 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:05.790735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5602bce0-90ee-438d-ae39-d3d5a2ef1c63 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:05.791479 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5602bce0-90ee-438d-ae39-d3d5a2ef1c63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:05.844127 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5602bce0-90ee-438d-ae39-d3d5a2ef1c63 None None] MFA Rules not processed for user `e28a0ba45389417ba8ae92376dba87ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:05.849586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5602bce0-90ee-438d-ae39-d3d5a2ef1c63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:05.942921 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1093/2173] 10.222.0.15 () {58 vars in 967 bytes} [Tue Sep 13 01:20:05 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:05.964319 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f98a316-2ad2-480f-ad49-22c9e407393b service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:05.966263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2f98a316-2ad2-480f-ad49-22c9e407393b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:05.979180 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:06.050368 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f98a316-2ad2-480f-ad49-22c9e407393b service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:06.055530 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28de24b6-87ee-4ae0-b872-e13c17d0a964 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d7d8ed4ede4ddca192fce0a497689d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:06.056144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28de24b6-87ee-4ae0-b872-e13c17d0a964 service swift] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:06.056422 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28de24b6-87ee-4ae0-b872-e13c17d0a964 service swift] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:06.056805 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28de24b6-87ee-4ae0-b872-e13c17d0a964 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:06.066317 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28de24b6-87ee-4ae0-b872-e13c17d0a964 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:06.067963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28de24b6-87ee-4ae0-b872-e13c17d0a964 service swift] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:06.076693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1081/2174] 10.222.0.15 () {60 vars in 1384 bytes} [Tue Sep 13 01:20:05 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 01:20:06.383096 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b4cf13-c058-49b5-b5a8-498d7bfb70cb None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:06.383716 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b4cf13-c058-49b5-b5a8-498d7bfb70cb None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:06.383716 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b4cf13-c058-49b5-b5a8-498d7bfb70cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:06.413715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-57b4cf13-c058-49b5-b5a8-498d7bfb70cb None None] MFA Rules not processed for user `e28a0ba45389417ba8ae92376dba87ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:06.414999 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-57b4cf13-c058-49b5-b5a8-498d7bfb70cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:06.424172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1094/2175] 10.222.0.15 () {58 vars in 1009 bytes} [Tue Sep 13 01:20:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:06.437451 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28de24b6-87ee-4ae0-b872-e13c17d0a964 service swift] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:06.451369 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28de24b6-87ee-4ae0-b872-e13c17d0a964 service swift] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:06.457603 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e534e282-b0ee-48b7-91ec-1eba49271ebc service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d7d8ed4ede4ddca192fce0a497689d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:06.458216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e534e282-b0ee-48b7-91ec-1eba49271ebc service swift] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:06.458495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e534e282-b0ee-48b7-91ec-1eba49271ebc service swift] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:06.458768 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e534e282-b0ee-48b7-91ec-1eba49271ebc service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:06.467425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e534e282-b0ee-48b7-91ec-1eba49271ebc service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:06.468999 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e534e282-b0ee-48b7-91ec-1eba49271ebc service swift] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:06.477343 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1082/2176] 10.222.0.15 () {60 vars in 1384 bytes} [Tue Sep 13 01:20:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 01:20:06.911708 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acb3905-dcff-4a21-b158-250dc70c1d74 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:06.912781 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acb3905-dcff-4a21-b158-250dc70c1d74 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:06.918611 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acb3905-dcff-4a21-b158-250dc70c1d74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:06.926408 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:06.956226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-3acb3905-dcff-4a21-b158-250dc70c1d74 None None] MFA Rules not processed for user `e28a0ba45389417ba8ae92376dba87ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:06.957422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3acb3905-dcff-4a21-b158-250dc70c1d74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:06.966133 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1095/2177] 10.222.0.15 () {58 vars in 967 bytes} [Tue Sep 13 01:20:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:06.980180 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e534e282-b0ee-48b7-91ec-1eba49271ebc service swift] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:07.000401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e534e282-b0ee-48b7-91ec-1eba49271ebc service swift] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:07.006467 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3b67dc-cf7a-44ac-9e39-7f705da6fe0e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d7d8ed4ede4ddca192fce0a497689d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:07.007270 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3b67dc-cf7a-44ac-9e39-7f705da6fe0e service swift] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:07.007784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3b67dc-cf7a-44ac-9e39-7f705da6fe0e service swift] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:07.008279 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3b67dc-cf7a-44ac-9e39-7f705da6fe0e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:07.010386 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acb3905-dcff-4a21-b158-250dc70c1d74 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:07.011892 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3acb3905-dcff-4a21-b158-250dc70c1d74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:07.020544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3b67dc-cf7a-44ac-9e39-7f705da6fe0e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:07.023326 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3b67dc-cf7a-44ac-9e39-7f705da6fe0e service swift] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:07.033327 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1083/2178] 10.222.0.15 () {60 vars in 1384 bytes} [Tue Sep 13 01:20:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 01:20:07.049903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acb3905-dcff-4a21-b158-250dc70c1d74 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:07.060006 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3d6a1f-3739-4dda-a550-9346b173a018 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:07.064546 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3d6a1f-3739-4dda-a550-9346b173a018 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:07.064840 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3d6a1f-3739-4dda-a550-9346b173a018 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:07.065107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3d6a1f-3739-4dda-a550-9346b173a018 admin admin] PATH_INFO: `/v3/users/14a6ec692df645c7878073d04fd97dd8` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:07.066504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3d6a1f-3739-4dda-a550-9346b173a018 admin admin] RBAC: Authorizing `identity:delete_user(user_id=14a6ec692df645c7878073d04fd97dd8)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:07.071091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3d6a1f-3739-4dda-a550-9346b173a018 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:07.184925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-8f3d6a1f-3739-4dda-a550-9346b173a018 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '14a6ec692df645c7878073d04fd97dd8'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:07.186878 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-8f3d6a1f-3739-4dda-a550-9346b173a018 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '14a6ec692df645c7878073d04fd97dd8'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:07.201598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-8f3d6a1f-3739-4dda-a550-9346b173a018 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '14a6ec692df645c7878073d04fd97dd8'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:07.202559 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-8f3d6a1f-3739-4dda-a550-9346b173a018 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '14a6ec692df645c7878073d04fd97dd8'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:07.238915 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-8f3d6a1f-3739-4dda-a550-9346b173a018 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '14a6ec692df645c7878073d04fd97dd8'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:07.254698 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1096/2179] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:20:07 2022] DELETE /identity/v3/users/14a6ec692df645c7878073d04fd97dd8 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:20:07.259408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa524f65-46d6-404e-a0dd-d1931a5cccbc None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:07.261103 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa524f65-46d6-404e-a0dd-d1931a5cccbc None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:07.261612 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa524f65-46d6-404e-a0dd-d1931a5cccbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:07.299884 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-aa524f65-46d6-404e-a0dd-d1931a5cccbc None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:07.302367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-aa524f65-46d6-404e-a0dd-d1931a5cccbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:07.422757 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1084/2180] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:07.439801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3d6a1f-3739-4dda-a550-9346b173a018 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:07.445155 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8f3d6a1f-3739-4dda-a550-9346b173a018 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:07.535565 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3d6a1f-3739-4dda-a550-9346b173a018 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:07.540444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0479a366-f5b3-43fa-88c9-02c66e7ec62d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:07.541792 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0479a366-f5b3-43fa-88c9-02c66e7ec62d service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:07.542254 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0479a366-f5b3-43fa-88c9-02c66e7ec62d service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:07.542728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0479a366-f5b3-43fa-88c9-02c66e7ec62d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:07.556472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0479a366-f5b3-43fa-88c9-02c66e7ec62d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:07.559564 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0479a366-f5b3-43fa-88c9-02c66e7ec62d service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:07.573684 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1097/2181] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:20:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:20:08.034101 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa524f65-46d6-404e-a0dd-d1931a5cccbc None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:08.036346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-aa524f65-46d6-404e-a0dd-d1931a5cccbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:08.077579 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa524f65-46d6-404e-a0dd-d1931a5cccbc None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:08.085095 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1156619-4ed0-46ea-9a11-6148c73c5430 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:08.085997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1156619-4ed0-46ea-9a11-6148c73c5430 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:08.086641 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1156619-4ed0-46ea-9a11-6148c73c5430 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:08.087124 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1156619-4ed0-46ea-9a11-6148c73c5430 admin admin] PATH_INFO: `/v3/projects/1fe54ba6fcc6433fb46a3afec1760029` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:08.088957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1156619-4ed0-46ea-9a11-6148c73c5430 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1fe54ba6fcc6433fb46a3afec1760029)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:08.092685 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1156619-4ed0-46ea-9a11-6148c73c5430 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:08.173903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d1156619-4ed0-46ea-9a11-6148c73c5430 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1fe54ba6fcc6433fb46a3afec1760029', 'request_id': 'req-d1156619-4ed0-46ea-9a11-6148c73c5430'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:08.184085 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d1156619-4ed0-46ea-9a11-6148c73c5430 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1fe54ba6fcc6433fb46a3afec1760029', 'request_id': 'req-d1156619-4ed0-46ea-9a11-6148c73c5430'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:08.186816 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:08.193624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d1156619-4ed0-46ea-9a11-6148c73c5430 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1fe54ba6fcc6433fb46a3afec1760029', 'request_id': 'req-d1156619-4ed0-46ea-9a11-6148c73c5430'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:08.216648 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d1156619-4ed0-46ea-9a11-6148c73c5430 admin admin] The token cache is being invalidate because project 1fe54ba6fcc6433fb46a3afec1760029 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:20:08.216648 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d1156619-4ed0-46ea-9a11-6148c73c5430 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:08.216648 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d1156619-4ed0-46ea-9a11-6148c73c5430 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:08.220095 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1085/2182] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:20:08 2022] DELETE /identity/v3/projects/1fe54ba6fcc6433fb46a3afec1760029 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:20:08.224088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0479a366-f5b3-43fa-88c9-02c66e7ec62d service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:08.234720 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0479a366-f5b3-43fa-88c9-02c66e7ec62d service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:08.331167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2126a58-8866-4948-b526-6a61479d1bc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:08.333677 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2126a58-8866-4948-b526-6a61479d1bc9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:08.334075 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2126a58-8866-4948-b526-6a61479d1bc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:08.334454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2126a58-8866-4948-b526-6a61479d1bc9 admin admin] PATH_INFO: `/v3/users/d8b28e817ef84fecb0d42aaad09c3004` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:08.336463 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2126a58-8866-4948-b526-6a61479d1bc9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d8b28e817ef84fecb0d42aaad09c3004)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:08.338561 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2126a58-8866-4948-b526-6a61479d1bc9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:08.359221 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:08.412433 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e2126a58-8866-4948-b526-6a61479d1bc9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd8b28e817ef84fecb0d42aaad09c3004'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:08.413421 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e2126a58-8866-4948-b526-6a61479d1bc9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd8b28e817ef84fecb0d42aaad09c3004'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:08.429292 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e2126a58-8866-4948-b526-6a61479d1bc9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd8b28e817ef84fecb0d42aaad09c3004'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:08.430173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e2126a58-8866-4948-b526-6a61479d1bc9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd8b28e817ef84fecb0d42aaad09c3004'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:08.452246 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e2126a58-8866-4948-b526-6a61479d1bc9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd8b28e817ef84fecb0d42aaad09c3004'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:08.472916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1098/2183] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:20:08 2022] DELETE /identity/v3/users/d8b28e817ef84fecb0d42aaad09c3004 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:20:08.976956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1156619-4ed0-46ea-9a11-6148c73c5430 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:08.978505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d1156619-4ed0-46ea-9a11-6148c73c5430 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:08.999552 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1156619-4ed0-46ea-9a11-6148c73c5430 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:09.076600 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b023d2-2263-41d1-9eda-5b15ac910242 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:09.077536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b023d2-2263-41d1-9eda-5b15ac910242 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:09.078250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b023d2-2263-41d1-9eda-5b15ac910242 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:09.078605 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b023d2-2263-41d1-9eda-5b15ac910242 admin admin] PATH_INFO: `/v3/projects/105a2bde9908451b81c509e7f45913a3` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:09.080070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b023d2-2263-41d1-9eda-5b15ac910242 admin admin] RBAC: Authorizing `identity:delete_project(project_id=105a2bde9908451b81c509e7f45913a3)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:09.082491 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b023d2-2263-41d1-9eda-5b15ac910242 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:09.158360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-94b023d2-2263-41d1-9eda-5b15ac910242 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '105a2bde9908451b81c509e7f45913a3', 'request_id': 'req-94b023d2-2263-41d1-9eda-5b15ac910242'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:09.168659 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-94b023d2-2263-41d1-9eda-5b15ac910242 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '105a2bde9908451b81c509e7f45913a3', 'request_id': 'req-94b023d2-2263-41d1-9eda-5b15ac910242'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:09.175810 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-94b023d2-2263-41d1-9eda-5b15ac910242 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '105a2bde9908451b81c509e7f45913a3', 'request_id': 'req-94b023d2-2263-41d1-9eda-5b15ac910242'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:09.197383 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-94b023d2-2263-41d1-9eda-5b15ac910242 admin admin] The token cache is being invalidate because project 105a2bde9908451b81c509e7f45913a3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:20:09.197707 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-94b023d2-2263-41d1-9eda-5b15ac910242 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:09.198465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-94b023d2-2263-41d1-9eda-5b15ac910242 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:09.200159 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1086/2184] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:20:08 2022] DELETE /identity/v3/projects/105a2bde9908451b81c509e7f45913a3 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:20:09.205393 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbe368a-9f74-4563-948d-4ba88d476c68 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:09.205675 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbe368a-9f74-4563-948d-4ba88d476c68 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:09.205947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbe368a-9f74-4563-948d-4ba88d476c68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:09.226360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9fbe368a-9f74-4563-948d-4ba88d476c68 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:09.227605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9fbe368a-9f74-4563-948d-4ba88d476c68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:09.304538 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1099/2185] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:09.313797 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d88a704-7067-4cad-a55f-5f95ff3bc7b9 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:09.314370 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d88a704-7067-4cad-a55f-5f95ff3bc7b9 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:09.314840 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d88a704-7067-4cad-a55f-5f95ff3bc7b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:09.341300 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7d88a704-7067-4cad-a55f-5f95ff3bc7b9 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:09.342793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7d88a704-7067-4cad-a55f-5f95ff3bc7b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:09.359047 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbe368a-9f74-4563-948d-4ba88d476c68 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:09.359833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1087/2186] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:20:09.375657 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbe368a-9f74-4563-948d-4ba88d476c68 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:09.380690 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2327480-5753-48a0-876e-53cd9254f378 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:09.381444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2327480-5753-48a0-876e-53cd9254f378 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:09.382293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2327480-5753-48a0-876e-53cd9254f378 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:09.382293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2327480-5753-48a0-876e-53cd9254f378 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:09.383035 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2327480-5753-48a0-876e-53cd9254f378 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:20:09.383842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2327480-5753-48a0-876e-53cd9254f378 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:09.385128 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2327480-5753-48a0-876e-53cd9254f378 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:09.405975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1100/2187] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:20:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:20:09.407220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d88a704-7067-4cad-a55f-5f95ff3bc7b9 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:09.424420 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d88a704-7067-4cad-a55f-5f95ff3bc7b9 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:09.435455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832d9b39-438b-4a21-a948-7a86f891ea67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:09.438005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832d9b39-438b-4a21-a948-7a86f891ea67 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:09.438379 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832d9b39-438b-4a21-a948-7a86f891ea67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:09.438755 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832d9b39-438b-4a21-a948-7a86f891ea67 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:09.439720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832d9b39-438b-4a21-a948-7a86f891ea67 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:09.449731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832d9b39-438b-4a21-a948-7a86f891ea67 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:09.516008 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1088/2188] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:20:09 2022] POST /identity/v3/projects => generated 381 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:20:09.517007 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2327480-5753-48a0-876e-53cd9254f378 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:09.539855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2327480-5753-48a0-876e-53cd9254f378 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:09.647127 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33ae021-04fe-4bf7-a35f-2d027b45c875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:09.647388 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33ae021-04fe-4bf7-a35f-2d027b45c875 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:09.647647 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33ae021-04fe-4bf7-a35f-2d027b45c875 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:09.647922 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33ae021-04fe-4bf7-a35f-2d027b45c875 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:09.648554 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33ae021-04fe-4bf7-a35f-2d027b45c875 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:09.650640 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33ae021-04fe-4bf7-a35f-2d027b45c875 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:09.688663 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832d9b39-438b-4a21-a948-7a86f891ea67 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:09.689101 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1101/2189] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:20:09 2022] POST /identity/v3/users => generated 291 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:20:09.700918 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832d9b39-438b-4a21-a948-7a86f891ea67 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:09.707751 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b789815-7814-425c-8908-77dbed11442c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:09.708381 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b789815-7814-425c-8908-77dbed11442c admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:09.708664 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b789815-7814-425c-8908-77dbed11442c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:09.708941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b789815-7814-425c-8908-77dbed11442c admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:09.709590 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b789815-7814-425c-8908-77dbed11442c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:09.711303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b789815-7814-425c-8908-77dbed11442c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:09.723682 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1089/2190] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:20:09 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:20:09.732900 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33ae021-04fe-4bf7-a35f-2d027b45c875 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:09.750489 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33ae021-04fe-4bf7-a35f-2d027b45c875 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:09.759985 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d392de10-2b02-4f8f-8329-47b2550ed47c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:09.760824 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d392de10-2b02-4f8f-8329-47b2550ed47c admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:09.761398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d392de10-2b02-4f8f-8329-47b2550ed47c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:09.761826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d392de10-2b02-4f8f-8329-47b2550ed47c admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:09.766483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d392de10-2b02-4f8f-8329-47b2550ed47c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:09.767126 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b789815-7814-425c-8908-77dbed11442c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:09.769105 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1b789815-7814-425c-8908-77dbed11442c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:09.771288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d392de10-2b02-4f8f-8329-47b2550ed47c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:09.776315 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:09.793407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b789815-7814-425c-8908-77dbed11442c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:09.801939 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53983087-4c07-448c-9bda-2af28c593b44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:09.802670 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53983087-4c07-448c-9bda-2af28c593b44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:09.803093 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53983087-4c07-448c-9bda-2af28c593b44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:09.803494 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53983087-4c07-448c-9bda-2af28c593b44 admin admin] PATH_INFO: `/v3/users/93409ebeeab6464abf34f26ebd37a2a9` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:09.804740 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1102/2191] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:20:09 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:20:09.809359 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d392de10-2b02-4f8f-8329-47b2550ed47c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:09.822979 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53983087-4c07-448c-9bda-2af28c593b44 admin admin] RBAC: Authorizing `identity:delete_user(user_id=93409ebeeab6464abf34f26ebd37a2a9)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:09.826002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d392de10-2b02-4f8f-8329-47b2550ed47c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:09.826907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53983087-4c07-448c-9bda-2af28c593b44 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:09.831788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86df943-745d-4bdc-b0dd-d5106287a042 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:09.832942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86df943-745d-4bdc-b0dd-d5106287a042 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:09.833414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86df943-745d-4bdc-b0dd-d5106287a042 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:09.833830 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86df943-745d-4bdc-b0dd-d5106287a042 admin admin] PATH_INFO: `/v3/projects/b60d18b7f10240aeaf459e267220343b/users/522a8bd8c1a84a279ce7c675021f0f23/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:09.858513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86df943-745d-4bdc-b0dd-d5106287a042 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b60d18b7f10240aeaf459e267220343b, user_id=522a8bd8c1a84a279ce7c675021f0f23, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:09.871876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86df943-745d-4bdc-b0dd-d5106287a042 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:09.923943 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=84,keystone:INSERT=3,keystone:DELETE=13 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:09.929760 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1103/2192] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:20:09 2022] PUT /identity/v3/projects/b60d18b7f10240aeaf459e267220343b/users/522a8bd8c1a84a279ce7c675021f0f23/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:20:09.937582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4d22af-6833-411d-ae65-c900edef908e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:09.937966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4d22af-6833-411d-ae65-c900edef908e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:09.938423 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4d22af-6833-411d-ae65-c900edef908e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:09.947468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-53983087-4c07-448c-9bda-2af28c593b44 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '93409ebeeab6464abf34f26ebd37a2a9'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:09.948292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-53983087-4c07-448c-9bda-2af28c593b44 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '93409ebeeab6464abf34f26ebd37a2a9'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:09.959990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-53983087-4c07-448c-9bda-2af28c593b44 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '93409ebeeab6464abf34f26ebd37a2a9'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:09.960784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-53983087-4c07-448c-9bda-2af28c593b44 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '93409ebeeab6464abf34f26ebd37a2a9'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:09.970215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-53983087-4c07-448c-9bda-2af28c593b44 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '93409ebeeab6464abf34f26ebd37a2a9'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:09.980187 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-dc4d22af-6833-411d-ae65-c900edef908e None None] MFA Rules not processed for user `522a8bd8c1a84a279ce7c675021f0f23`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:09.981655 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-dc4d22af-6833-411d-ae65-c900edef908e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:09.998157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1090/2193] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:20:09 2022] DELETE /identity/v3/users/93409ebeeab6464abf34f26ebd37a2a9 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:20:10.003509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302b33dc-a11b-4bfc-a547-1944ce32942b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:10.003917 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302b33dc-a11b-4bfc-a547-1944ce32942b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:10.004291 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302b33dc-a11b-4bfc-a547-1944ce32942b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:10.031810 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-302b33dc-a11b-4bfc-a547-1944ce32942b None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:10.033462 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-302b33dc-a11b-4bfc-a547-1944ce32942b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:10.137728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1104/2194] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:09 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:10.159883 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4d22af-6833-411d-ae65-c900edef908e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:10.161366 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-dc4d22af-6833-411d-ae65-c900edef908e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:10.165263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1091/2195] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:10.179604 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302b33dc-a11b-4bfc-a547-1944ce32942b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:10.181997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-302b33dc-a11b-4bfc-a547-1944ce32942b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:10.259654 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4d22af-6833-411d-ae65-c900edef908e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:10.292697 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302b33dc-a11b-4bfc-a547-1944ce32942b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:10.299119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4440297-00d5-4784-a21f-03a60b391522 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:10.300150 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4440297-00d5-4784-a21f-03a60b391522 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:10.300584 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4440297-00d5-4784-a21f-03a60b391522 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:10.301022 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4440297-00d5-4784-a21f-03a60b391522 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:10.315031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4440297-00d5-4784-a21f-03a60b391522 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:10.317148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4440297-00d5-4784-a21f-03a60b391522 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:10.345133 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1092/2196] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:20:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:20:10.381712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da7cb2b-4fed-4900-9190-bf74a7f708a5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:10.383060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da7cb2b-4fed-4900-9190-bf74a7f708a5 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:10.383365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da7cb2b-4fed-4900-9190-bf74a7f708a5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:10.383652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da7cb2b-4fed-4900-9190-bf74a7f708a5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:10.487882 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:10.550451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da7cb2b-4fed-4900-9190-bf74a7f708a5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:10.552093 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da7cb2b-4fed-4900-9190-bf74a7f708a5 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:10.566732 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1105/2197] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:20:10 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:20:10.590276 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4440297-00d5-4784-a21f-03a60b391522 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:10.591842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da7cb2b-4fed-4900-9190-bf74a7f708a5 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:10.592293 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f4440297-00d5-4784-a21f-03a60b391522 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:10.593743 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7da7cb2b-4fed-4900-9190-bf74a7f708a5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:10.612521 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4440297-00d5-4784-a21f-03a60b391522 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:10.619472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da7cb2b-4fed-4900-9190-bf74a7f708a5 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:10.621990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c166706-fe3f-4253-94b7-1de34a756ca0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:10.623027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c166706-fe3f-4253-94b7-1de34a756ca0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:10.623577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c166706-fe3f-4253-94b7-1de34a756ca0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:10.624065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c166706-fe3f-4253-94b7-1de34a756ca0 admin admin] PATH_INFO: `/v3/users/522a8bd8c1a84a279ce7c675021f0f23` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:10.625862 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c166706-fe3f-4253-94b7-1de34a756ca0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=522a8bd8c1a84a279ce7c675021f0f23)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:10.629175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c166706-fe3f-4253-94b7-1de34a756ca0 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:10.630333 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e578488-f274-402b-94d3-281891e876fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:10.631092 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e578488-f274-402b-94d3-281891e876fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:10.631466 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e578488-f274-402b-94d3-281891e876fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:10.631812 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e578488-f274-402b-94d3-281891e876fc admin admin] PATH_INFO: `/v3/projects/b0422bd9612349b8a0cbcae6f1a588b7` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:10.633686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e578488-f274-402b-94d3-281891e876fc admin admin] RBAC: Authorizing `identity:delete_project(project_id=b0422bd9612349b8a0cbcae6f1a588b7)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:10.646567 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e578488-f274-402b-94d3-281891e876fc admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:10.713549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2e578488-f274-402b-94d3-281891e876fc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b0422bd9612349b8a0cbcae6f1a588b7', 'request_id': 'req-2e578488-f274-402b-94d3-281891e876fc'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:10.716535 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4c166706-fe3f-4253-94b7-1de34a756ca0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '522a8bd8c1a84a279ce7c675021f0f23'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:10.717285 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4c166706-fe3f-4253-94b7-1de34a756ca0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '522a8bd8c1a84a279ce7c675021f0f23'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:10.720066 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2e578488-f274-402b-94d3-281891e876fc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b0422bd9612349b8a0cbcae6f1a588b7', 'request_id': 'req-2e578488-f274-402b-94d3-281891e876fc'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:10.720859 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:10.728695 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4c166706-fe3f-4253-94b7-1de34a756ca0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '522a8bd8c1a84a279ce7c675021f0f23'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:10.729328 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4c166706-fe3f-4253-94b7-1de34a756ca0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '522a8bd8c1a84a279ce7c675021f0f23'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:10.732035 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2e578488-f274-402b-94d3-281891e876fc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b0422bd9612349b8a0cbcae6f1a588b7', 'request_id': 'req-2e578488-f274-402b-94d3-281891e876fc'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:10.747728 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4c166706-fe3f-4253-94b7-1de34a756ca0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '522a8bd8c1a84a279ce7c675021f0f23'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:10.749493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2e578488-f274-402b-94d3-281891e876fc admin admin] The token cache is being invalidate because project b0422bd9612349b8a0cbcae6f1a588b7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:20:10.749786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2e578488-f274-402b-94d3-281891e876fc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:10.750424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2e578488-f274-402b-94d3-281891e876fc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:10.752806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1106/2198] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:20:10 2022] DELETE /identity/v3/projects/b0422bd9612349b8a0cbcae6f1a588b7 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:20:10.759918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fd7dc5-ba03-414c-b7d6-dcca6f5e9d90 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:10.760465 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fd7dc5-ba03-414c-b7d6-dcca6f5e9d90 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:10.760973 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fd7dc5-ba03-414c-b7d6-dcca6f5e9d90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:10.774534 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1093/2199] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:20:10 2022] DELETE /identity/v3/users/522a8bd8c1a84a279ce7c675021f0f23 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:20:10.779521 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8bc039-66fb-4733-a319-f0b343b4318d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:10.780015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8bc039-66fb-4733-a319-f0b343b4318d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:10.780444 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8bc039-66fb-4733-a319-f0b343b4318d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:10.792689 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-20fd7dc5-ba03-414c-b7d6-dcca6f5e9d90 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:10.794912 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-20fd7dc5-ba03-414c-b7d6-dcca6f5e9d90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:10.824504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7d8bc039-66fb-4733-a319-f0b343b4318d None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:10.825753 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7d8bc039-66fb-4733-a319-f0b343b4318d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:10.936479 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1107/2200] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:10.938105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b55a17c-0357-4a56-83ac-9698af77fe92 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:10.938352 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b55a17c-0357-4a56-83ac-9698af77fe92 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:10.938638 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b55a17c-0357-4a56-83ac-9698af77fe92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:10.964985 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4b55a17c-0357-4a56-83ac-9698af77fe92 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:10.969194 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4b55a17c-0357-4a56-83ac-9698af77fe92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:10.975287 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1094/2201] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:10.983385 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1108/2202] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:10.983827 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8bc039-66fb-4733-a319-f0b343b4318d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:10.985226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7d8bc039-66fb-4733-a319-f0b343b4318d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:11.001639 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b55a17c-0357-4a56-83ac-9698af77fe92 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:11.014049 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b55a17c-0357-4a56-83ac-9698af77fe92 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:11.022594 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819b784c-2db6-4bcd-9b35-6450191744b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:11.023286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819b784c-2db6-4bcd-9b35-6450191744b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:11.023563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819b784c-2db6-4bcd-9b35-6450191744b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:11.026184 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819b784c-2db6-4bcd-9b35-6450191744b5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:11.026881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819b784c-2db6-4bcd-9b35-6450191744b5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:20:11.027862 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819b784c-2db6-4bcd-9b35-6450191744b5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:11.040035 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819b784c-2db6-4bcd-9b35-6450191744b5 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:11.084631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1109/2203] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:20:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:20:11.084990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8bc039-66fb-4733-a319-f0b343b4318d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:11.085672 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819b784c-2db6-4bcd-9b35-6450191744b5 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:11.090154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8477ba50-aaa1-422b-8338-6a3ba31eea90 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:11.091067 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8477ba50-aaa1-422b-8338-6a3ba31eea90 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:11.091449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8477ba50-aaa1-422b-8338-6a3ba31eea90 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:11.092797 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8477ba50-aaa1-422b-8338-6a3ba31eea90 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:11.102714 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819b784c-2db6-4bcd-9b35-6450191744b5 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:11.110004 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8477ba50-aaa1-422b-8338-6a3ba31eea90 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:11.112660 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a936c711-4e7a-4cf4-8468-f6e43e689d01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:11.113359 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a936c711-4e7a-4cf4-8468-f6e43e689d01 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:11.113762 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a936c711-4e7a-4cf4-8468-f6e43e689d01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:11.114106 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a936c711-4e7a-4cf4-8468-f6e43e689d01 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:11.114884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a936c711-4e7a-4cf4-8468-f6e43e689d01 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:11.117123 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8477ba50-aaa1-422b-8338-6a3ba31eea90 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:11.118224 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a936c711-4e7a-4cf4-8468-f6e43e689d01 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:11.133479 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1095/2204] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:20:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:20:11.162730 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1110/2205] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:20:11 2022] POST /identity/v3/projects => generated 377 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:20:11.167125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8477ba50-aaa1-422b-8338-6a3ba31eea90 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:11.178502 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8477ba50-aaa1-422b-8338-6a3ba31eea90 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:11.210931 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3a94a2-51e4-40df-95fe-7d9b5e475547 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:11.211461 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3a94a2-51e4-40df-95fe-7d9b5e475547 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:11.211976 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3a94a2-51e4-40df-95fe-7d9b5e475547 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:11.262657 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:11.280658 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece36f5e-6bab-4d43-b79e-92deb2b54e76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:11.280902 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-0f3a94a2-51e4-40df-95fe-7d9b5e475547 None None] MFA Rules not processed for user `776733af5b6a4115b2a11d67666a379d`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:11.281424 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece36f5e-6bab-4d43-b79e-92deb2b54e76 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:11.281720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece36f5e-6bab-4d43-b79e-92deb2b54e76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:11.281980 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece36f5e-6bab-4d43-b79e-92deb2b54e76 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:11.283384 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece36f5e-6bab-4d43-b79e-92deb2b54e76 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:11.285804 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece36f5e-6bab-4d43-b79e-92deb2b54e76 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:11.288246 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0f3a94a2-51e4-40df-95fe-7d9b5e475547 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:11.304869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:DELETE=3,keystone:SELECT=94,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:11.316618 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1096/2206] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:20:11 2022] POST /identity/v3/users => generated 289 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:20:11.317668 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece36f5e-6bab-4d43-b79e-92deb2b54e76 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:11.345245 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece36f5e-6bab-4d43-b79e-92deb2b54e76 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:11.359570 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9bfcd4-c183-458b-a58c-12d3bed780fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:11.360451 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9bfcd4-c183-458b-a58c-12d3bed780fc admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:11.360733 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9bfcd4-c183-458b-a58c-12d3bed780fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:11.361009 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9bfcd4-c183-458b-a58c-12d3bed780fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:11.361669 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9bfcd4-c183-458b-a58c-12d3bed780fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:11.365584 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9bfcd4-c183-458b-a58c-12d3bed780fc admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:11.381778 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1097/2207] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:20:11 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:20:11.385171 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9bfcd4-c183-458b-a58c-12d3bed780fc admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:11.397237 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9bfcd4-c183-458b-a58c-12d3bed780fc admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:11.408617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a806d917-06a6-4d5e-bc57-b763561c7188 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:11.409526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a806d917-06a6-4d5e-bc57-b763561c7188 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:11.409526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a806d917-06a6-4d5e-bc57-b763561c7188 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:11.409843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a806d917-06a6-4d5e-bc57-b763561c7188 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:11.410604 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a806d917-06a6-4d5e-bc57-b763561c7188 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:11.412554 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a806d917-06a6-4d5e-bc57-b763561c7188 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:11.432221 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1111/2208] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:11 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:11.448475 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3a94a2-51e4-40df-95fe-7d9b5e475547 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:11.450455 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0f3a94a2-51e4-40df-95fe-7d9b5e475547 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:11.458522 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1098/2209] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:20:11 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:20:11.460145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a806d917-06a6-4d5e-bc57-b763561c7188 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:11.477319 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a806d917-06a6-4d5e-bc57-b763561c7188 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:11.491968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a09c379-1a9e-47af-b716-b74bedc65ca6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:11.492723 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a09c379-1a9e-47af-b716-b74bedc65ca6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:11.493063 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a09c379-1a9e-47af-b716-b74bedc65ca6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:11.494505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a09c379-1a9e-47af-b716-b74bedc65ca6 admin admin] PATH_INFO: `/v3/projects/66cdd4ec03ea4fbf959ede65bb96f6ca/users/2842b222fe6b4821a670d13ec8febf1a/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:11.524455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a09c379-1a9e-47af-b716-b74bedc65ca6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=66cdd4ec03ea4fbf959ede65bb96f6ca, user_id=2842b222fe6b4821a670d13ec8febf1a, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:11.527167 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a09c379-1a9e-47af-b716-b74bedc65ca6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:11.567455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1099/2210] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:20:11 2022] PUT /identity/v3/projects/66cdd4ec03ea4fbf959ede65bb96f6ca/users/2842b222fe6b4821a670d13ec8febf1a/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:20:11.572580 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9f6fdd-fcc0-4d96-bddc-1f4362b15054 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:11.573287 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9f6fdd-fcc0-4d96-bddc-1f4362b15054 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:11.574900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9f6fdd-fcc0-4d96-bddc-1f4362b15054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:11.585868 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3a94a2-51e4-40df-95fe-7d9b5e475547 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:11.631916 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8e9f6fdd-fcc0-4d96-bddc-1f4362b15054 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:11.633163 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8e9f6fdd-fcc0-4d96-bddc-1f4362b15054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:11.758515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60958a3-c40f-4ed6-b9f6-ac0a757d3a5e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:11.758515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60958a3-c40f-4ed6-b9f6-ac0a757d3a5e service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:11.758515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60958a3-c40f-4ed6-b9f6-ac0a757d3a5e service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:11.758515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60958a3-c40f-4ed6-b9f6-ac0a757d3a5e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:11.799364 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1100/2211] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:20:11.800503 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9f6fdd-fcc0-4d96-bddc-1f4362b15054 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:11.809367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8e9f6fdd-fcc0-4d96-bddc-1f4362b15054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:11.819803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:11.835357 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9f6fdd-fcc0-4d96-bddc-1f4362b15054 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:11.843723 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105260f0-da07-4b47-8a3c-d1278eacdcd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:11.847119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105260f0-da07-4b47-8a3c-d1278eacdcd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:11.847624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105260f0-da07-4b47-8a3c-d1278eacdcd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:11.848504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105260f0-da07-4b47-8a3c-d1278eacdcd6 admin admin] PATH_INFO: `/v3/projects/b60d18b7f10240aeaf459e267220343b` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:11.851983 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60958a3-c40f-4ed6-b9f6-ac0a757d3a5e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:11.852704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105260f0-da07-4b47-8a3c-d1278eacdcd6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b60d18b7f10240aeaf459e267220343b)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:11.854085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60958a3-c40f-4ed6-b9f6-ac0a757d3a5e service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:11.859238 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105260f0-da07-4b47-8a3c-d1278eacdcd6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:11.870938 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1112/2212] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:20:11 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:20:11.872088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60958a3-c40f-4ed6-b9f6-ac0a757d3a5e service nova] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:11.873973 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f60958a3-c40f-4ed6-b9f6-ac0a757d3a5e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:11.969083 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-105260f0-da07-4b47-8a3c-d1278eacdcd6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b60d18b7f10240aeaf459e267220343b', 'request_id': 'req-105260f0-da07-4b47-8a3c-d1278eacdcd6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:11.971124 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60958a3-c40f-4ed6-b9f6-ac0a757d3a5e service nova] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:11.983102 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-105260f0-da07-4b47-8a3c-d1278eacdcd6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b60d18b7f10240aeaf459e267220343b', 'request_id': 'req-105260f0-da07-4b47-8a3c-d1278eacdcd6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:11.991011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-105260f0-da07-4b47-8a3c-d1278eacdcd6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b60d18b7f10240aeaf459e267220343b', 'request_id': 'req-105260f0-da07-4b47-8a3c-d1278eacdcd6'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:12.013634 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-105260f0-da07-4b47-8a3c-d1278eacdcd6 admin admin] The token cache is being invalidate because project b60d18b7f10240aeaf459e267220343b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:20:12.014644 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-105260f0-da07-4b47-8a3c-d1278eacdcd6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:12.016407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-105260f0-da07-4b47-8a3c-d1278eacdcd6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:12.018767 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1101/2213] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:20:11 2022] DELETE /identity/v3/projects/b60d18b7f10240aeaf459e267220343b => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:20:12.025288 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a607c89-5e95-4dea-be33-3aaad71f6f94 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:12.027026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a607c89-5e95-4dea-be33-3aaad71f6f94 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:12.027026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a607c89-5e95-4dea-be33-3aaad71f6f94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:12.052548 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-1a607c89-5e95-4dea-be33-3aaad71f6f94 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:12.053799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1a607c89-5e95-4dea-be33-3aaad71f6f94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:12.064771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c83a0dc-d671-4441-a64b-038efaee2c4b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:12.065384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c83a0dc-d671-4441-a64b-038efaee2c4b service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:12.065661 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c83a0dc-d671-4441-a64b-038efaee2c4b service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:12.065929 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c83a0dc-d671-4441-a64b-038efaee2c4b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:12.080172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9c83a0dc-d671-4441-a64b-038efaee2c4b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:12.093311 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:12.144455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1102/2214] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:12.152635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3500fc-8010-4ab6-99d2-29392588c143 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:12.152971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3500fc-8010-4ab6-99d2-29392588c143 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:12.153392 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3500fc-8010-4ab6-99d2-29392588c143 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:12.180028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ca3500fc-8010-4ab6-99d2-29392588c143 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:12.182004 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ca3500fc-8010-4ab6-99d2-29392588c143 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:12.190094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c83a0dc-d671-4441-a64b-038efaee2c4b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:12.192355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c83a0dc-d671-4441-a64b-038efaee2c4b service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:12.204787 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1103/2215] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:20:12.207690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3500fc-8010-4ab6-99d2-29392588c143 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:12.218881 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3500fc-8010-4ab6-99d2-29392588c143 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:12.224265 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1113/2216] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:20:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:20:12.233996 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa63b7b4-36f2-4947-9b56-285830768401 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:12.236843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa63b7b4-36f2-4947-9b56-285830768401 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:12.237146 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa63b7b4-36f2-4947-9b56-285830768401 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:12.237430 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa63b7b4-36f2-4947-9b56-285830768401 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:12.238114 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa63b7b4-36f2-4947-9b56-285830768401 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:20:12.238420 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa63b7b4-36f2-4947-9b56-285830768401 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:12.240224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa63b7b4-36f2-4947-9b56-285830768401 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:12.255312 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1104/2217] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:20:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:20:12.259933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c83a0dc-d671-4441-a64b-038efaee2c4b service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:12.270191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c83a0dc-d671-4441-a64b-038efaee2c4b service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:12.276253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53662bba-ac54-4758-8cc8-7f1ee93df638 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:12.316702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53662bba-ac54-4758-8cc8-7f1ee93df638 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:12.319514 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53662bba-ac54-4758-8cc8-7f1ee93df638 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:12.321546 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53662bba-ac54-4758-8cc8-7f1ee93df638 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:12.322442 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53662bba-ac54-4758-8cc8-7f1ee93df638 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:12.325726 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53662bba-ac54-4758-8cc8-7f1ee93df638 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:12.392792 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1114/2218] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:20:12 2022] POST /identity/v3/projects => generated 373 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:20:12.396972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa63b7b4-36f2-4947-9b56-285830768401 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:12.413063 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa63b7b4-36f2-4947-9b56-285830768401 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:12.540756 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b431950e-e3c8-4546-8e91-414f4a404e31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:12.542095 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b431950e-e3c8-4546-8e91-414f4a404e31 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:12.544143 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b431950e-e3c8-4546-8e91-414f4a404e31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:12.544696 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b431950e-e3c8-4546-8e91-414f4a404e31 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:12.545660 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b431950e-e3c8-4546-8e91-414f4a404e31 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:12.550513 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b431950e-e3c8-4546-8e91-414f4a404e31 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:12.585975 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1105/2219] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:20:12 2022] POST /identity/v3/users => generated 287 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:20:12.591008 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53662bba-ac54-4758-8cc8-7f1ee93df638 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:12.620115 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53662bba-ac54-4758-8cc8-7f1ee93df638 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:12.625332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652fb3f8-0909-4fe0-aad9-ce45d257796c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:12.626060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652fb3f8-0909-4fe0-aad9-ce45d257796c admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:12.626366 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652fb3f8-0909-4fe0-aad9-ce45d257796c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:12.626721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652fb3f8-0909-4fe0-aad9-ce45d257796c admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:12.627371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652fb3f8-0909-4fe0-aad9-ce45d257796c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:12.629345 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652fb3f8-0909-4fe0-aad9-ce45d257796c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:12.644443 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1115/2220] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:20:12 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:20:12.648313 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b431950e-e3c8-4546-8e91-414f4a404e31 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:12.662409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b431950e-e3c8-4546-8e91-414f4a404e31 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:12.669728 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7dab787-52e4-4bc3-b180-c649944d3e64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:12.670497 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7dab787-52e4-4bc3-b180-c649944d3e64 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:12.670780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7dab787-52e4-4bc3-b180-c649944d3e64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:12.671083 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7dab787-52e4-4bc3-b180-c649944d3e64 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:12.671889 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7dab787-52e4-4bc3-b180-c649944d3e64 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:12.673526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7dab787-52e4-4bc3-b180-c649944d3e64 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:12.690769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1106/2221] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:20:12 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:20:12.691938 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652fb3f8-0909-4fe0-aad9-ce45d257796c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:12.719012 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652fb3f8-0909-4fe0-aad9-ce45d257796c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:12.724747 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7ab25a-9d0d-4a5f-a008-a20c34ec97be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:12.725744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7ab25a-9d0d-4a5f-a008-a20c34ec97be admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:12.726208 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7ab25a-9d0d-4a5f-a008-a20c34ec97be admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:12.726681 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7ab25a-9d0d-4a5f-a008-a20c34ec97be admin admin] PATH_INFO: `/v3/projects/e2cf3f7032d9458d983f92e3939bd6da/users/e6b1269f9928414e9391e46cf56aef76/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:12.744319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:12.757963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7ab25a-9d0d-4a5f-a008-a20c34ec97be admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2cf3f7032d9458d983f92e3939bd6da, user_id=e6b1269f9928414e9391e46cf56aef76, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:12.761042 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7ab25a-9d0d-4a5f-a008-a20c34ec97be admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:12.800179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1116/2222] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:20:12 2022] PUT /identity/v3/projects/e2cf3f7032d9458d983f92e3939bd6da/users/e6b1269f9928414e9391e46cf56aef76/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:20:12.813132 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959401a3-4fac-46d1-bb00-13ca8e7fa9bf None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:12.813638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959401a3-4fac-46d1-bb00-13ca8e7fa9bf None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:12.814142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959401a3-4fac-46d1-bb00-13ca8e7fa9bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:12.877625 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-959401a3-4fac-46d1-bb00-13ca8e7fa9bf None None] MFA Rules not processed for user `e6b1269f9928414e9391e46cf56aef76`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:12.879018 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-959401a3-4fac-46d1-bb00-13ca8e7fa9bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:13.012614 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1107/2223] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:12 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:13.025237 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7ab25a-9d0d-4a5f-a008-a20c34ec97be admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:13.028784 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9d7ab25a-9d0d-4a5f-a008-a20c34ec97be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:13.134283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7ab25a-9d0d-4a5f-a008-a20c34ec97be admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:13.140727 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6a4fb7-35e5-4384-aab3-71cb2aa58e34 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:13.141687 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6a4fb7-35e5-4384-aab3-71cb2aa58e34 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:13.142074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6a4fb7-35e5-4384-aab3-71cb2aa58e34 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:13.142449 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6a4fb7-35e5-4384-aab3-71cb2aa58e34 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:13.161310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6a4fb7-35e5-4384-aab3-71cb2aa58e34 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:13.163330 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6a4fb7-35e5-4384-aab3-71cb2aa58e34 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:13.183032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1117/2224] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:20:13 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:20:13.212977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959401a3-4fac-46d1-bb00-13ca8e7fa9bf None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:13.253605 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:13.318740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959401a3-4fac-46d1-bb00-13ca8e7fa9bf None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:13.340546 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d0cdad-be99-4f72-8c6c-99d0d9ccf8d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:13.341290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d0cdad-be99-4f72-8c6c-99d0d9ccf8d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:13.341591 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d0cdad-be99-4f72-8c6c-99d0d9ccf8d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:13.341872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d0cdad-be99-4f72-8c6c-99d0d9ccf8d9 admin admin] PATH_INFO: `/v3/users/e6b1269f9928414e9391e46cf56aef76` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:13.344339 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d0cdad-be99-4f72-8c6c-99d0d9ccf8d9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e6b1269f9928414e9391e46cf56aef76)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:13.346452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d0cdad-be99-4f72-8c6c-99d0d9ccf8d9 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:13.410838 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-85d0cdad-be99-4f72-8c6c-99d0d9ccf8d9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e6b1269f9928414e9391e46cf56aef76'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:13.411507 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-85d0cdad-be99-4f72-8c6c-99d0d9ccf8d9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e6b1269f9928414e9391e46cf56aef76'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:13.428952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-85d0cdad-be99-4f72-8c6c-99d0d9ccf8d9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e6b1269f9928414e9391e46cf56aef76'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:13.429683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-85d0cdad-be99-4f72-8c6c-99d0d9ccf8d9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e6b1269f9928414e9391e46cf56aef76'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:13.450582 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-85d0cdad-be99-4f72-8c6c-99d0d9ccf8d9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e6b1269f9928414e9391e46cf56aef76'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:13.506458 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1108/2225] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:20:13 2022] DELETE /identity/v3/users/e6b1269f9928414e9391e46cf56aef76 => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:20:13.508327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4232fba-a0e6-47a2-9faf-5ff28e4a6ce1 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:13.508634 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4232fba-a0e6-47a2-9faf-5ff28e4a6ce1 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:13.508998 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4232fba-a0e6-47a2-9faf-5ff28e4a6ce1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:13.538295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b4232fba-a0e6-47a2-9faf-5ff28e4a6ce1 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:13.544437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b4232fba-a0e6-47a2-9faf-5ff28e4a6ce1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:13.698110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1118/2226] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:13.700122 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d0cdad-be99-4f72-8c6c-99d0d9ccf8d9 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:13.702399 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-85d0cdad-be99-4f72-8c6c-99d0d9ccf8d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:13.810256 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d0cdad-be99-4f72-8c6c-99d0d9ccf8d9 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:13.820889 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bca7f9-e75f-4b61-9986-89f88cce210f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:13.822888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bca7f9-e75f-4b61-9986-89f88cce210f service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:13.822888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bca7f9-e75f-4b61-9986-89f88cce210f service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:13.822888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bca7f9-e75f-4b61-9986-89f88cce210f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:13.837292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bca7f9-e75f-4b61-9986-89f88cce210f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:13.839209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bca7f9-e75f-4b61-9986-89f88cce210f service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:13.856523 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1109/2227] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:20:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:20:14.321900 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4232fba-a0e6-47a2-9faf-5ff28e4a6ce1 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:14.323481 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b4232fba-a0e6-47a2-9faf-5ff28e4a6ce1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:14.354269 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4232fba-a0e6-47a2-9faf-5ff28e4a6ce1 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:14.367063 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05346890-b7e4-43a2-b968-5b1fc35af642 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:14.367977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05346890-b7e4-43a2-b968-5b1fc35af642 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:14.370183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05346890-b7e4-43a2-b968-5b1fc35af642 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:14.371607 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05346890-b7e4-43a2-b968-5b1fc35af642 admin admin] PATH_INFO: `/v3/projects/e2cf3f7032d9458d983f92e3939bd6da` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:14.373298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05346890-b7e4-43a2-b968-5b1fc35af642 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2cf3f7032d9458d983f92e3939bd6da)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:14.375977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05346890-b7e4-43a2-b968-5b1fc35af642 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:14.468876 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-05346890-b7e4-43a2-b968-5b1fc35af642 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2cf3f7032d9458d983f92e3939bd6da', 'request_id': 'req-05346890-b7e4-43a2-b968-5b1fc35af642'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:14.476531 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-05346890-b7e4-43a2-b968-5b1fc35af642 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2cf3f7032d9458d983f92e3939bd6da', 'request_id': 'req-05346890-b7e4-43a2-b968-5b1fc35af642'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:14.486335 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-05346890-b7e4-43a2-b968-5b1fc35af642 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2cf3f7032d9458d983f92e3939bd6da', 'request_id': 'req-05346890-b7e4-43a2-b968-5b1fc35af642'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:14.505651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-05346890-b7e4-43a2-b968-5b1fc35af642 admin admin] The token cache is being invalidate because project e2cf3f7032d9458d983f92e3939bd6da was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:20:14.506284 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-05346890-b7e4-43a2-b968-5b1fc35af642 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:14.507631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-05346890-b7e4-43a2-b968-5b1fc35af642 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:20:14.510073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1119/2228] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:20:14 2022] DELETE /identity/v3/projects/e2cf3f7032d9458d983f92e3939bd6da => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:20:14.515692 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ea2999-fa23-428d-a6b2-5ff8cf54bf60 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:14.516664 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ea2999-fa23-428d-a6b2-5ff8cf54bf60 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:14.517254 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ea2999-fa23-428d-a6b2-5ff8cf54bf60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:14.554763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fed94f-8d24-4115-9c21-993a46ce08ff None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:14.555190 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fed94f-8d24-4115-9c21-993a46ce08ff None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:14.555548 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fed94f-8d24-4115-9c21-993a46ce08ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:14.627576 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-27fed94f-8d24-4115-9c21-993a46ce08ff None None] MFA Rules not processed for user `776733af5b6a4115b2a11d67666a379d`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:14.628018 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-77ea2999-fa23-428d-a6b2-5ff8cf54bf60 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:14.633678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-27fed94f-8d24-4115-9c21-993a46ce08ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:14.635619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-77ea2999-fa23-428d-a6b2-5ff8cf54bf60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:14.663508 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:14.773027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:14.782451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1120/2229] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:14 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:14.802646 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1110/2230] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 289 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:14.813165 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8dc382-0373-462b-ba5f-6db7adbecdf0 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:14.813165 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8dc382-0373-462b-ba5f-6db7adbecdf0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:14.813477 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8dc382-0373-462b-ba5f-6db7adbecdf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:14.847392 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-0e8dc382-0373-462b-ba5f-6db7adbecdf0 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:14.849958 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0e8dc382-0373-462b-ba5f-6db7adbecdf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:14.862888 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1121/2231] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:14.867354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ea2999-fa23-428d-a6b2-5ff8cf54bf60 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:14.880533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ea2999-fa23-428d-a6b2-5ff8cf54bf60 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:14.887570 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99183994-8bad-4656-91f1-d4c663c6a460 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:14.888272 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99183994-8bad-4656-91f1-d4c663c6a460 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:14.888561 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99183994-8bad-4656-91f1-d4c663c6a460 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:14.888838 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99183994-8bad-4656-91f1-d4c663c6a460 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:14.889599 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99183994-8bad-4656-91f1-d4c663c6a460 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:20:14.889971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99183994-8bad-4656-91f1-d4c663c6a460 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:14.892420 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99183994-8bad-4656-91f1-d4c663c6a460 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:14.908676 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1111/2232] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:20:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:20:14.913398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8dc382-0373-462b-ba5f-6db7adbecdf0 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:14.947175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8dc382-0373-462b-ba5f-6db7adbecdf0 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:14.961230 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64876f3e-9645-45f5-8d03-9a37ce2cf2ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:14.961860 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64876f3e-9645-45f5-8d03-9a37ce2cf2ec admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:14.962648 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64876f3e-9645-45f5-8d03-9a37ce2cf2ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:14.962648 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64876f3e-9645-45f5-8d03-9a37ce2cf2ec admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:14.963074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64876f3e-9645-45f5-8d03-9a37ce2cf2ec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:14.965070 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64876f3e-9645-45f5-8d03-9a37ce2cf2ec admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:15.006235 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1122/2233] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:20:14 2022] POST /identity/v3/projects => generated 365 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:20:15.010431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99183994-8bad-4656-91f1-d4c663c6a460 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:15.028186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99183994-8bad-4656-91f1-d4c663c6a460 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:15.139587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d025acd-7485-4f1d-ba97-c12a22e1013f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:15.140498 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d025acd-7485-4f1d-ba97-c12a22e1013f admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:15.140949 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d025acd-7485-4f1d-ba97-c12a22e1013f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:15.141422 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d025acd-7485-4f1d-ba97-c12a22e1013f admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:15.142359 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d025acd-7485-4f1d-ba97-c12a22e1013f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:15.144954 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d025acd-7485-4f1d-ba97-c12a22e1013f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:15.195930 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1112/2234] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:20:15 2022] POST /identity/v3/users => generated 283 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:20:15.196746 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64876f3e-9645-45f5-8d03-9a37ce2cf2ec admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:15.212465 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64876f3e-9645-45f5-8d03-9a37ce2cf2ec admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:15.225419 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e18ca69-88ed-4166-ab1f-dade29e3fcfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:15.227525 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e18ca69-88ed-4166-ab1f-dade29e3fcfd admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:15.228534 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e18ca69-88ed-4166-ab1f-dade29e3fcfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:15.229255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e18ca69-88ed-4166-ab1f-dade29e3fcfd admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:15.230620 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e18ca69-88ed-4166-ab1f-dade29e3fcfd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:15.232982 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e18ca69-88ed-4166-ab1f-dade29e3fcfd admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:15.253134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1123/2235] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:20:15 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:20:15.258015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d025acd-7485-4f1d-ba97-c12a22e1013f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:15.275957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d025acd-7485-4f1d-ba97-c12a22e1013f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:15.294643 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2b208c-fd9f-41a0-b895-807bce1a9c37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:15.295397 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2b208c-fd9f-41a0-b895-807bce1a9c37 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:15.296479 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2b208c-fd9f-41a0-b895-807bce1a9c37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:15.297622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2b208c-fd9f-41a0-b895-807bce1a9c37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:15.298380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2b208c-fd9f-41a0-b895-807bce1a9c37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:15.302743 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2b208c-fd9f-41a0-b895-807bce1a9c37 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:15.334187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1113/2236] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:20:15 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:20:15.339337 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e18ca69-88ed-4166-ab1f-dade29e3fcfd admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:15.360167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e18ca69-88ed-4166-ab1f-dade29e3fcfd admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:15.375456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0700dd10-4ceb-45a8-864a-960439c6bdb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:15.376152 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0700dd10-4ceb-45a8-864a-960439c6bdb3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:15.376438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0700dd10-4ceb-45a8-864a-960439c6bdb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:15.376718 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0700dd10-4ceb-45a8-864a-960439c6bdb3 admin admin] PATH_INFO: `/v3/projects/1bacdd426f05441f9944633295773ff6/users/7e0b93fc70ee4f38a000b6ad15f66b57/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:15.426884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0700dd10-4ceb-45a8-864a-960439c6bdb3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1bacdd426f05441f9944633295773ff6, user_id=7e0b93fc70ee4f38a000b6ad15f66b57, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:15.429723 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0700dd10-4ceb-45a8-864a-960439c6bdb3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:15.485867 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1124/2237] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:20:15 2022] PUT /identity/v3/projects/1bacdd426f05441f9944633295773ff6/users/7e0b93fc70ee4f38a000b6ad15f66b57/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:20:15.497260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c5bdd7-2b72-4e94-ad75-3738b95457b7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:15.497496 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c5bdd7-2b72-4e94-ad75-3738b95457b7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:15.497765 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c5bdd7-2b72-4e94-ad75-3738b95457b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:15.590096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-53c5bdd7-2b72-4e94-ad75-3738b95457b7 None None] MFA Rules not processed for user `7e0b93fc70ee4f38a000b6ad15f66b57`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:15.595393 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-53c5bdd7-2b72-4e94-ad75-3738b95457b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:15.723222 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1114/2238] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:15 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:15.737247 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0700dd10-4ceb-45a8-864a-960439c6bdb3 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:15.739624 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0700dd10-4ceb-45a8-864a-960439c6bdb3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:15.832266 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0700dd10-4ceb-45a8-864a-960439c6bdb3 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:15.838156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac09cd9-d9c8-4540-9755-12c9c7a14290 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:15.839272 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac09cd9-d9c8-4540-9755-12c9c7a14290 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:15.840445 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac09cd9-d9c8-4540-9755-12c9c7a14290 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:15.840773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac09cd9-d9c8-4540-9755-12c9c7a14290 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:15.852493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac09cd9-d9c8-4540-9755-12c9c7a14290 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:15.854175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac09cd9-d9c8-4540-9755-12c9c7a14290 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:15.879284 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1125/2239] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:20:15 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:20:20.054564 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d219314-605b-4b6e-abe7-c5402c88bb1f None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:20.055102 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d219314-605b-4b6e-abe7-c5402c88bb1f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:20.057031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d219314-605b-4b6e-abe7-c5402c88bb1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:20.120184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-5d219314-605b-4b6e-abe7-c5402c88bb1f None None] MFA Rules not processed for user `2842b222fe6b4821a670d13ec8febf1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:20.121882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5d219314-605b-4b6e-abe7-c5402c88bb1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:20.177863 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:20.226365 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1115/2240] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:20 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:20.231164 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac09cd9-d9c8-4540-9755-12c9c7a14290 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:20.233237 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-eac09cd9-d9c8-4540-9755-12c9c7a14290 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:20.302508 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:20.309192 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac09cd9-d9c8-4540-9755-12c9c7a14290 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:20.316918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bc1eff-091b-4b18-b875-07bf9339be47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:20.320414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bc1eff-091b-4b18-b875-07bf9339be47 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:20.320715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bc1eff-091b-4b18-b875-07bf9339be47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:20.320986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bc1eff-091b-4b18-b875-07bf9339be47 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:20.323371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bc1eff-091b-4b18-b875-07bf9339be47 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:20.326005 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bc1eff-091b-4b18-b875-07bf9339be47 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:20.359677 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1126/2241] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:20:20 2022] POST /identity/v3/projects => generated 379 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:20:20.363780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d219314-605b-4b6e-abe7-c5402c88bb1f None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:20.377228 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d219314-605b-4b6e-abe7-c5402c88bb1f None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:20.456109 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf84cfb7-26c0-45c3-9a72-8e9d3c9646e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:20.456721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf84cfb7-26c0-45c3-9a72-8e9d3c9646e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:20.457046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf84cfb7-26c0-45c3-9a72-8e9d3c9646e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:20.457268 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf84cfb7-26c0-45c3-9a72-8e9d3c9646e6 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:20.457987 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf84cfb7-26c0-45c3-9a72-8e9d3c9646e6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:20.460223 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf84cfb7-26c0-45c3-9a72-8e9d3c9646e6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:20.488367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1116/2242] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:20:20 2022] POST /identity/v3/users => generated 296 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:20:20.489701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bc1eff-091b-4b18-b875-07bf9339be47 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:20.500463 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bc1eff-091b-4b18-b875-07bf9339be47 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:20.506672 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f33bc6-f160-49c6-819d-30f6b8eff675 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:20.507339 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f33bc6-f160-49c6-819d-30f6b8eff675 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:20.507714 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f33bc6-f160-49c6-819d-30f6b8eff675 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:20.508004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f33bc6-f160-49c6-819d-30f6b8eff675 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:20.508689 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f33bc6-f160-49c6-819d-30f6b8eff675 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:20.511123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f33bc6-f160-49c6-819d-30f6b8eff675 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:20.524789 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1127/2243] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:20:20 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:20:20.529549 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf84cfb7-26c0-45c3-9a72-8e9d3c9646e6 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:20.550173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf84cfb7-26c0-45c3-9a72-8e9d3c9646e6 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:20.557136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458c2fd2-822b-4304-9598-7811cd9e123a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:20.559141 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458c2fd2-822b-4304-9598-7811cd9e123a admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:20.559901 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458c2fd2-822b-4304-9598-7811cd9e123a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:20.560320 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458c2fd2-822b-4304-9598-7811cd9e123a admin admin] PATH_INFO: `/v3/projects/01cdb885e3894f3eb1d6947137d35b67/users/0d17e633508f4d0eb77a81bb840ce7b9/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:20.588696 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458c2fd2-822b-4304-9598-7811cd9e123a admin admin] RBAC: Authorizing `identity:create_grant(project_id=01cdb885e3894f3eb1d6947137d35b67, user_id=0d17e633508f4d0eb77a81bb840ce7b9, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:20.592243 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458c2fd2-822b-4304-9598-7811cd9e123a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:20.607337 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1117/2244] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:20:20 2022] PUT /identity/v3/projects/01cdb885e3894f3eb1d6947137d35b67/users/0d17e633508f4d0eb77a81bb840ce7b9/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:20:20.613024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f33bc6-f160-49c6-819d-30f6b8eff675 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:20.623698 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f33bc6-f160-49c6-819d-30f6b8eff675 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:20.691010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64effe9-2cf2-4861-ab12-c687497a4094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:20.691729 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64effe9-2cf2-4861-ab12-c687497a4094 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:20.691972 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64effe9-2cf2-4861-ab12-c687497a4094 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:20.692250 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64effe9-2cf2-4861-ab12-c687497a4094 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:20.692934 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64effe9-2cf2-4861-ab12-c687497a4094 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:20.694570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64effe9-2cf2-4861-ab12-c687497a4094 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:20.705800 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1128/2245] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:20:20 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:20:20.710405 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458c2fd2-822b-4304-9598-7811cd9e123a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:20.724113 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458c2fd2-822b-4304-9598-7811cd9e123a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:20.730734 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79045e42-e615-4b71-be2b-55e6849b1864 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:20.731599 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79045e42-e615-4b71-be2b-55e6849b1864 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:20.731883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79045e42-e615-4b71-be2b-55e6849b1864 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:20.732153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79045e42-e615-4b71-be2b-55e6849b1864 admin admin] PATH_INFO: `/v3/projects/01cdb885e3894f3eb1d6947137d35b67/users/0d17e633508f4d0eb77a81bb840ce7b9/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:20.734615 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79045e42-e615-4b71-be2b-55e6849b1864 admin admin] RBAC: Authorizing `identity:create_grant(project_id=01cdb885e3894f3eb1d6947137d35b67, user_id=0d17e633508f4d0eb77a81bb840ce7b9, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:20.738729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79045e42-e615-4b71-be2b-55e6849b1864 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:20.752765 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1118/2246] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:20:20 2022] PUT /identity/v3/projects/01cdb885e3894f3eb1d6947137d35b67/users/0d17e633508f4d0eb77a81bb840ce7b9/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:20:27.197791 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d2834c-bf6c-4e23-9368-927dc908c3cd None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:27.199021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d2834c-bf6c-4e23-9368-927dc908c3cd None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:27.199313 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d2834c-bf6c-4e23-9368-927dc908c3cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:27.287078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-74d2834c-bf6c-4e23-9368-927dc908c3cd None None] MFA Rules not processed for user `0d17e633508f4d0eb77a81bb840ce7b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:27.288813 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-74d2834c-bf6c-4e23-9368-927dc908c3cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:27.428795 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1129/2247] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:27 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:27.436542 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79045e42-e615-4b71-be2b-55e6849b1864 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:27.441260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-79045e42-e615-4b71-be2b-55e6849b1864 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:27.564154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79045e42-e615-4b71-be2b-55e6849b1864 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:27.569704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781bff96-c07d-4308-9044-80248adcd286 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:27.570510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781bff96-c07d-4308-9044-80248adcd286 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:27.570948 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781bff96-c07d-4308-9044-80248adcd286 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:27.571450 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781bff96-c07d-4308-9044-80248adcd286 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:27.602042 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:27.708484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781bff96-c07d-4308-9044-80248adcd286 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:27.719503 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781bff96-c07d-4308-9044-80248adcd286 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:27.747517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1119/2248] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:20:27 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:20:37.404299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=75,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:37.732988 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=17 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:41.519060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebd9914-5caf-45d2-b5ad-2a03161f9cdd None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:41.519547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebd9914-5caf-45d2-b5ad-2a03161f9cdd None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:41.519720 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebd9914-5caf-45d2-b5ad-2a03161f9cdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:41.539907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-eebd9914-5caf-45d2-b5ad-2a03161f9cdd None None] MFA Rules not processed for user `0c5dcfa5794843a5896fa051a4554e1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:41.541091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-eebd9914-5caf-45d2-b5ad-2a03161f9cdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:41.608442 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1130/2249] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:41 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:47.798504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781bff96-c07d-4308-9044-80248adcd286 service nova] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:47.799990 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-781bff96-c07d-4308-9044-80248adcd286 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:47.876728 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781bff96-c07d-4308-9044-80248adcd286 service nova] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:47.880752 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e1f0ca-0073-43ae-8508-a701dd950b79 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:47.881634 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e1f0ca-0073-43ae-8508-a701dd950b79 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:47.881919 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e1f0ca-0073-43ae-8508-a701dd950b79 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:47.882205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e1f0ca-0073-43ae-8508-a701dd950b79 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:47.886406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-02e1f0ca-0073-43ae-8508-a701dd950b79 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:47.981926 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e1f0ca-0073-43ae-8508-a701dd950b79 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:47.984064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e1f0ca-0073-43ae-8508-a701dd950b79 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:47.995412 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1120/2250] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:20:47 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:20:51.601331 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=25 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:20:53.947608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fd44c0-fd23-4551-afc5-da1cae2fc20c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:53.948229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fd44c0-fd23-4551-afc5-da1cae2fc20c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:53.948229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fd44c0-fd23-4551-afc5-da1cae2fc20c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:53.968833 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e6fd44c0-fd23-4551-afc5-da1cae2fc20c None None] MFA Rules not processed for user `2842b222fe6b4821a670d13ec8febf1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:53.969953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e6fd44c0-fd23-4551-afc5-da1cae2fc20c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:53.976775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1131/2251] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:20:53 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:57.711837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a630df37-9bac-479c-8aa2-081c62dd743c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:57.712346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a630df37-9bac-479c-8aa2-081c62dd743c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:57.712461 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a630df37-9bac-479c-8aa2-081c62dd743c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:57.739025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-a630df37-9bac-479c-8aa2-081c62dd743c None None] MFA Rules not processed for user `e28a0ba45389417ba8ae92376dba87ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:20:57.740402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a630df37-9bac-479c-8aa2-081c62dd743c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:57.815158 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1121/2252] 10.222.0.15 () {58 vars in 1009 bytes} [Tue Sep 13 01:20:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:20:57.827757 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fd44c0-fd23-4551-afc5-da1cae2fc20c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:20:57.829320 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e6fd44c0-fd23-4551-afc5-da1cae2fc20c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:20:57.929434 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fd44c0-fd23-4551-afc5-da1cae2fc20c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:20:57.937236 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1114af-7b60-4b95-9b84-b04299610462 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d7d8ed4ede4ddca192fce0a497689d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:20:57.938054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1114af-7b60-4b95-9b84-b04299610462 service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:20:57.938683 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1114af-7b60-4b95-9b84-b04299610462 service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:20:57.939165 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1114af-7b60-4b95-9b84-b04299610462 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:20:57.958947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1114af-7b60-4b95-9b84-b04299610462 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:20:57.960816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1114af-7b60-4b95-9b84-b04299610462 service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:20:57.972505 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1132/2253] 10.222.0.15 () {60 vars in 1384 bytes} [Tue Sep 13 01:20:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 01:21:03.419959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a630df37-9bac-479c-8aa2-081c62dd743c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:03.535235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:21:03.543156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a630df37-9bac-479c-8aa2-081c62dd743c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:03.548896 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc040c8d-d01c-4763-9c55-87ec812fc999 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:03.549673 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc040c8d-d01c-4763-9c55-87ec812fc999 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:03.550018 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc040c8d-d01c-4763-9c55-87ec812fc999 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:03.550474 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc040c8d-d01c-4763-9c55-87ec812fc999 admin admin] PATH_INFO: `/v3/users/7e0b93fc70ee4f38a000b6ad15f66b57` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:03.552806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc040c8d-d01c-4763-9c55-87ec812fc999 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7e0b93fc70ee4f38a000b6ad15f66b57)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:03.554887 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc040c8d-d01c-4763-9c55-87ec812fc999 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:03.642129 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-fc040c8d-d01c-4763-9c55-87ec812fc999 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7e0b93fc70ee4f38a000b6ad15f66b57'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:03.642796 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-fc040c8d-d01c-4763-9c55-87ec812fc999 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7e0b93fc70ee4f38a000b6ad15f66b57'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:03.671036 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-fc040c8d-d01c-4763-9c55-87ec812fc999 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7e0b93fc70ee4f38a000b6ad15f66b57'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:03.672145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-fc040c8d-d01c-4763-9c55-87ec812fc999 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7e0b93fc70ee4f38a000b6ad15f66b57'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:03.689803 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-fc040c8d-d01c-4763-9c55-87ec812fc999 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7e0b93fc70ee4f38a000b6ad15f66b57'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:03.717612 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1122/2254] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:21:03 2022] DELETE /identity/v3/users/7e0b93fc70ee4f38a000b6ad15f66b57 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:21:03.721779 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc1b9da-533f-4536-ab6f-68b10332faec None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:03.722317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc1b9da-533f-4536-ab6f-68b10332faec None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:03.722317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc1b9da-533f-4536-ab6f-68b10332faec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:03.742403 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8cc1b9da-533f-4536-ab6f-68b10332faec None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:21:03.744643 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8cc1b9da-533f-4536-ab6f-68b10332faec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:03.823483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1133/2255] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:21:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:21:03.833162 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc040c8d-d01c-4763-9c55-87ec812fc999 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:03.835091 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fc040c8d-d01c-4763-9c55-87ec812fc999 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:03.930613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc040c8d-d01c-4763-9c55-87ec812fc999 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:03.937352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5a80aa-cd3d-4656-a7d8-95b3f8a69553 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:03.937781 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5a80aa-cd3d-4656-a7d8-95b3f8a69553 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:03.938067 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5a80aa-cd3d-4656-a7d8-95b3f8a69553 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:03.938342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5a80aa-cd3d-4656-a7d8-95b3f8a69553 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:03.949645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5a80aa-cd3d-4656-a7d8-95b3f8a69553 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:03.952161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5a80aa-cd3d-4656-a7d8-95b3f8a69553 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:03.965264 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1123/2256] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:21:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:21:04.309191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc1b9da-533f-4536-ab6f-68b10332faec None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:04.311052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8cc1b9da-533f-4536-ab6f-68b10332faec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:04.327211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc1b9da-533f-4536-ab6f-68b10332faec None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:04.331987 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939c61eb-7848-49e3-aa0a-c933f729c501 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:04.332776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939c61eb-7848-49e3-aa0a-c933f729c501 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:04.332902 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939c61eb-7848-49e3-aa0a-c933f729c501 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:04.333169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939c61eb-7848-49e3-aa0a-c933f729c501 admin admin] PATH_INFO: `/v3/projects/1bacdd426f05441f9944633295773ff6` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:04.334483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939c61eb-7848-49e3-aa0a-c933f729c501 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1bacdd426f05441f9944633295773ff6)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:04.336437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939c61eb-7848-49e3-aa0a-c933f729c501 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:04.452847 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-939c61eb-7848-49e3-aa0a-c933f729c501 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1bacdd426f05441f9944633295773ff6', 'request_id': 'req-939c61eb-7848-49e3-aa0a-c933f729c501'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:04.462986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-939c61eb-7848-49e3-aa0a-c933f729c501 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1bacdd426f05441f9944633295773ff6', 'request_id': 'req-939c61eb-7848-49e3-aa0a-c933f729c501'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:04.473400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-939c61eb-7848-49e3-aa0a-c933f729c501 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1bacdd426f05441f9944633295773ff6', 'request_id': 'req-939c61eb-7848-49e3-aa0a-c933f729c501'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:04.501172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-939c61eb-7848-49e3-aa0a-c933f729c501 admin admin] The token cache is being invalidate because project 1bacdd426f05441f9944633295773ff6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:21:04.501995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-939c61eb-7848-49e3-aa0a-c933f729c501 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:04.503491 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-939c61eb-7848-49e3-aa0a-c933f729c501 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:04.506761 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1134/2257] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:21:04 2022] DELETE /identity/v3/projects/1bacdd426f05441f9944633295773ff6 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:21:04.512530 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd081f35-8392-4d34-a5c3-d785f1aef3a2 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:04.513005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd081f35-8392-4d34-a5c3-d785f1aef3a2 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:04.513134 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd081f35-8392-4d34-a5c3-d785f1aef3a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:04.549221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-cd081f35-8392-4d34-a5c3-d785f1aef3a2 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:21:04.550507 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cd081f35-8392-4d34-a5c3-d785f1aef3a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:04.686508 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1124/2258] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:21:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:21:04.687925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b50d5a8-e82d-4683-b399-f7820e9129bb None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:04.688217 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b50d5a8-e82d-4683-b399-f7820e9129bb None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:04.688497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b50d5a8-e82d-4683-b399-f7820e9129bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:04.717862 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:21:04.751376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5b50d5a8-e82d-4683-b399-f7820e9129bb None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:21:04.753190 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5b50d5a8-e82d-4683-b399-f7820e9129bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:04.775350 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1135/2259] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:21:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:21:04.779731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd081f35-8392-4d34-a5c3-d785f1aef3a2 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:04.798035 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd081f35-8392-4d34-a5c3-d785f1aef3a2 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:04.805458 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f90c9d-78a1-4912-9a1b-571f54bd4f90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:04.807147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f90c9d-78a1-4912-9a1b-571f54bd4f90 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:04.807629 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f90c9d-78a1-4912-9a1b-571f54bd4f90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:04.808198 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f90c9d-78a1-4912-9a1b-571f54bd4f90 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:04.809131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f90c9d-78a1-4912-9a1b-571f54bd4f90 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:21:04.810100 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f90c9d-78a1-4912-9a1b-571f54bd4f90 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:04.812063 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f90c9d-78a1-4912-9a1b-571f54bd4f90 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:04.850949 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1125/2260] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:21:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:21:04.855278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b50d5a8-e82d-4683-b399-f7820e9129bb None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:04.892361 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b50d5a8-e82d-4683-b399-f7820e9129bb None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:04.904589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27634f0-8dba-4ba6-be64-a2d8482e26d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:04.905245 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27634f0-8dba-4ba6-be64-a2d8482e26d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:04.905511 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27634f0-8dba-4ba6-be64-a2d8482e26d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:04.905789 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27634f0-8dba-4ba6-be64-a2d8482e26d3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:04.907874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27634f0-8dba-4ba6-be64-a2d8482e26d3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:04.917094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27634f0-8dba-4ba6-be64-a2d8482e26d3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:04.966205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1136/2261] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:21:04 2022] POST /identity/v3/projects => generated 371 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:21:04.970789 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f90c9d-78a1-4912-9a1b-571f54bd4f90 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:04.978339 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:21:04.985561 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f90c9d-78a1-4912-9a1b-571f54bd4f90 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:05.125025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9905a0-a692-4d6f-8d66-db4de461c94c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:05.125748 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9905a0-a692-4d6f-8d66-db4de461c94c admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:05.126035 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9905a0-a692-4d6f-8d66-db4de461c94c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:05.126323 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9905a0-a692-4d6f-8d66-db4de461c94c admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:05.126982 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9905a0-a692-4d6f-8d66-db4de461c94c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:05.131154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9905a0-a692-4d6f-8d66-db4de461c94c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:05.175641 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1126/2262] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:21:04 2022] POST /identity/v3/users => generated 286 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:21:05.176573 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27634f0-8dba-4ba6-be64-a2d8482e26d3 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:05.188655 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27634f0-8dba-4ba6-be64-a2d8482e26d3 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:05.193583 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2789f5-cc04-4e80-bdf0-f4f6120be54e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:05.194424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2789f5-cc04-4e80-bdf0-f4f6120be54e admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:05.194709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2789f5-cc04-4e80-bdf0-f4f6120be54e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:05.195022 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2789f5-cc04-4e80-bdf0-f4f6120be54e admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:05.195762 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2789f5-cc04-4e80-bdf0-f4f6120be54e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:05.197889 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2789f5-cc04-4e80-bdf0-f4f6120be54e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:05.217668 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1137/2263] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:21:05 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:21:05.218611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9905a0-a692-4d6f-8d66-db4de461c94c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:05.231307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9905a0-a692-4d6f-8d66-db4de461c94c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:05.237565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243028c6-2efe-4626-9f1a-e45e9e13bef4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:05.238239 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243028c6-2efe-4626-9f1a-e45e9e13bef4 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:05.238528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243028c6-2efe-4626-9f1a-e45e9e13bef4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:05.238806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243028c6-2efe-4626-9f1a-e45e9e13bef4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:05.239483 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243028c6-2efe-4626-9f1a-e45e9e13bef4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:05.241091 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243028c6-2efe-4626-9f1a-e45e9e13bef4 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:05.253508 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1127/2264] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:21:05 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:21:05.257038 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2789f5-cc04-4e80-bdf0-f4f6120be54e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:05.281524 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2789f5-cc04-4e80-bdf0-f4f6120be54e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:05.286459 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c759970-5805-4324-8733-43808f1e7daf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:05.287412 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c759970-5805-4324-8733-43808f1e7daf admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:05.287699 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c759970-5805-4324-8733-43808f1e7daf admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:05.287978 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c759970-5805-4324-8733-43808f1e7daf admin admin] PATH_INFO: `/v3/projects/6edfb14c2e0848ec857d22cbd731748e/users/fdb4cec8e9eb436fb4321a9c22d4b4f4/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:05.322300 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c759970-5805-4324-8733-43808f1e7daf admin admin] RBAC: Authorizing `identity:create_grant(project_id=6edfb14c2e0848ec857d22cbd731748e, user_id=fdb4cec8e9eb436fb4321a9c22d4b4f4, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:05.328457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c759970-5805-4324-8733-43808f1e7daf admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:05.361188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1138/2265] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:21:05 2022] PUT /identity/v3/projects/6edfb14c2e0848ec857d22cbd731748e/users/fdb4cec8e9eb436fb4321a9c22d4b4f4/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:21:05.381017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7772bad-d4d3-4624-85c3-7216a12eac33 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:05.381935 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7772bad-d4d3-4624-85c3-7216a12eac33 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:05.382231 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7772bad-d4d3-4624-85c3-7216a12eac33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:05.438891 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-a7772bad-d4d3-4624-85c3-7216a12eac33 None None] MFA Rules not processed for user `fdb4cec8e9eb436fb4321a9c22d4b4f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:21:05.438891 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a7772bad-d4d3-4624-85c3-7216a12eac33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:05.564293 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1128/2266] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:21:05 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:21:05.583295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c759970-5805-4324-8733-43808f1e7daf admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:05.585033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2c759970-5805-4324-8733-43808f1e7daf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:05.649031 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c759970-5805-4324-8733-43808f1e7daf admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:05.653346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354430fd-a0d9-4560-99f7-914464a5406c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:05.653960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354430fd-a0d9-4560-99f7-914464a5406c service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:05.654276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354430fd-a0d9-4560-99f7-914464a5406c service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:05.654575 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354430fd-a0d9-4560-99f7-914464a5406c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:05.670403 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354430fd-a0d9-4560-99f7-914464a5406c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:05.676368 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354430fd-a0d9-4560-99f7-914464a5406c service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:05.687762 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1139/2267] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:21:05 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:21:05.725776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7772bad-d4d3-4624-85c3-7216a12eac33 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:05.808223 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7772bad-d4d3-4624-85c3-7216a12eac33 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:05.815335 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380b810d-5857-485c-b076-a758dc4f8e89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:05.816103 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380b810d-5857-485c-b076-a758dc4f8e89 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:05.816481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380b810d-5857-485c-b076-a758dc4f8e89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:05.816869 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380b810d-5857-485c-b076-a758dc4f8e89 admin admin] PATH_INFO: `/v3/users/fdb4cec8e9eb436fb4321a9c22d4b4f4` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:05.818610 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380b810d-5857-485c-b076-a758dc4f8e89 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fdb4cec8e9eb436fb4321a9c22d4b4f4)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:05.820671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380b810d-5857-485c-b076-a758dc4f8e89 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:05.838248 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:21:05.887206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-380b810d-5857-485c-b076-a758dc4f8e89 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fdb4cec8e9eb436fb4321a9c22d4b4f4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:05.887815 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-380b810d-5857-485c-b076-a758dc4f8e89 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fdb4cec8e9eb436fb4321a9c22d4b4f4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:05.899454 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-380b810d-5857-485c-b076-a758dc4f8e89 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fdb4cec8e9eb436fb4321a9c22d4b4f4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:05.900729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-380b810d-5857-485c-b076-a758dc4f8e89 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fdb4cec8e9eb436fb4321a9c22d4b4f4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:05.908734 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-380b810d-5857-485c-b076-a758dc4f8e89 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fdb4cec8e9eb436fb4321a9c22d4b4f4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:05.937969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67209af9-ffa5-4bc3-8bc8-f06ddc9c15f5 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:05.938553 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67209af9-ffa5-4bc3-8bc8-f06ddc9c15f5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:05.938660 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67209af9-ffa5-4bc3-8bc8-f06ddc9c15f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:05.941625 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1129/2268] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:21:05 2022] DELETE /identity/v3/users/fdb4cec8e9eb436fb4321a9c22d4b4f4 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:21:05.961587 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-67209af9-ffa5-4bc3-8bc8-f06ddc9c15f5 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:21:05.963631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-67209af9-ffa5-4bc3-8bc8-f06ddc9c15f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:06.035060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1140/2269] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:21:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:21:06.047145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380b810d-5857-485c-b076-a758dc4f8e89 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:06.049414 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-380b810d-5857-485c-b076-a758dc4f8e89 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:06.195961 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380b810d-5857-485c-b076-a758dc4f8e89 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:06.220653 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb70b25-3dd9-41cd-adc9-01f76165ec32 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:06.224221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb70b25-3dd9-41cd-adc9-01f76165ec32 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:06.224999 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb70b25-3dd9-41cd-adc9-01f76165ec32 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:06.227240 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb70b25-3dd9-41cd-adc9-01f76165ec32 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:06.261118 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb70b25-3dd9-41cd-adc9-01f76165ec32 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:06.267203 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb70b25-3dd9-41cd-adc9-01f76165ec32 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:06.300688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1130/2270] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:21:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 254 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:21:06.744482 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67209af9-ffa5-4bc3-8bc8-f06ddc9c15f5 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:06.746609 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-67209af9-ffa5-4bc3-8bc8-f06ddc9c15f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:06.773509 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67209af9-ffa5-4bc3-8bc8-f06ddc9c15f5 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:06.778654 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84376e5d-0f0c-4ce6-97f6-5f440c06cfa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:06.780214 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84376e5d-0f0c-4ce6-97f6-5f440c06cfa0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:06.780563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84376e5d-0f0c-4ce6-97f6-5f440c06cfa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:06.780911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84376e5d-0f0c-4ce6-97f6-5f440c06cfa0 admin admin] PATH_INFO: `/v3/projects/6edfb14c2e0848ec857d22cbd731748e` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:06.782662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84376e5d-0f0c-4ce6-97f6-5f440c06cfa0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6edfb14c2e0848ec857d22cbd731748e)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:06.786623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84376e5d-0f0c-4ce6-97f6-5f440c06cfa0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:06.842776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:21:06.851216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-84376e5d-0f0c-4ce6-97f6-5f440c06cfa0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6edfb14c2e0848ec857d22cbd731748e', 'request_id': 'req-84376e5d-0f0c-4ce6-97f6-5f440c06cfa0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:06.860819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-84376e5d-0f0c-4ce6-97f6-5f440c06cfa0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6edfb14c2e0848ec857d22cbd731748e', 'request_id': 'req-84376e5d-0f0c-4ce6-97f6-5f440c06cfa0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:06.870024 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-84376e5d-0f0c-4ce6-97f6-5f440c06cfa0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6edfb14c2e0848ec857d22cbd731748e', 'request_id': 'req-84376e5d-0f0c-4ce6-97f6-5f440c06cfa0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:06.885744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-84376e5d-0f0c-4ce6-97f6-5f440c06cfa0 admin admin] The token cache is being invalidate because project 6edfb14c2e0848ec857d22cbd731748e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:21:06.886060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-84376e5d-0f0c-4ce6-97f6-5f440c06cfa0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:06.886842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-84376e5d-0f0c-4ce6-97f6-5f440c06cfa0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:21:06.890927 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1141/2271] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:21:06 2022] DELETE /identity/v3/projects/6edfb14c2e0848ec857d22cbd731748e => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:21:13.713684 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f235b67-ac58-47aa-b851-b55f0850ebd8 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:13.714258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f235b67-ac58-47aa-b851-b55f0850ebd8 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:13.714426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f235b67-ac58-47aa-b851-b55f0850ebd8 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:13.715829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1131/2272] 10.222.0.15 () {54 vars in 921 bytes} [Tue Sep 13 01:21:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:21:13.722570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074e7b3b-940b-45c9-af41-da2a907b57e5 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:13.723156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074e7b3b-940b-45c9-af41-da2a907b57e5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:13.723156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074e7b3b-940b-45c9-af41-da2a907b57e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:13.743174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-074e7b3b-940b-45c9-af41-da2a907b57e5 None None] MFA Rules not processed for user `2f2da4d2d3604d16b58dc25b8c17c49a`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:21:13.744367 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-074e7b3b-940b-45c9-af41-da2a907b57e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:13.814129 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1142/2273] 10.222.0.15 () {58 vars in 1018 bytes} [Tue Sep 13 01:21:13 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:21:13.826069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f235b67-ac58-47aa-b851-b55f0850ebd8 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:13.827883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7f235b67-ac58-47aa-b851-b55f0850ebd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:13.906594 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f235b67-ac58-47aa-b851-b55f0850ebd8 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:13.910159 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea55ae86-ae94-4824-8bfa-5eab4c3a62b5 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eef98bcf2699461d9a7f3c3bffe82f34', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:13.910757 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea55ae86-ae94-4824-8bfa-5eab4c3a62b5 service glance] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:13.911068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea55ae86-ae94-4824-8bfa-5eab4c3a62b5 service glance] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:13.911342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea55ae86-ae94-4824-8bfa-5eab4c3a62b5 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:13.921891 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea55ae86-ae94-4824-8bfa-5eab4c3a62b5 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:13.923357 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea55ae86-ae94-4824-8bfa-5eab4c3a62b5 service glance] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:13.933970 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1132/2274] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:21:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:21:14.002525 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962c9cd8-71a8-402c-b431-a4816a1c10df None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:14.002970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962c9cd8-71a8-402c-b431-a4816a1c10df None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:14.003250 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962c9cd8-71a8-402c-b431-a4816a1c10df None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:14.004621 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1143/2275] 10.222.0.15 () {54 vars in 921 bytes} [Tue Sep 13 01:21:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:21:14.009597 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9f4f40-a3f8-40bf-a6ef-4cc0be7a748d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:14.009876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9f4f40-a3f8-40bf-a6ef-4cc0be7a748d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:14.010156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9f4f40-a3f8-40bf-a6ef-4cc0be7a748d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:14.030616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8d9f4f40-a3f8-40bf-a6ef-4cc0be7a748d None None] MFA Rules not processed for user `09cd263f72484733adaaa0850ebfa609`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:21:14.031865 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8d9f4f40-a3f8-40bf-a6ef-4cc0be7a748d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:14.044347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:21:14.085646 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1133/2276] 10.222.0.15 () {58 vars in 1018 bytes} [Tue Sep 13 01:21:14 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:21:14.097727 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962c9cd8-71a8-402c-b431-a4816a1c10df None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:14.099452 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-962c9cd8-71a8-402c-b431-a4816a1c10df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:14.112917 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962c9cd8-71a8-402c-b431-a4816a1c10df None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:14.116972 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ea3164-b777-4c39-82aa-3aade451f221 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:14.117644 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ea3164-b777-4c39-82aa-3aade451f221 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:14.117958 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ea3164-b777-4c39-82aa-3aade451f221 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:14.118266 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ea3164-b777-4c39-82aa-3aade451f221 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:14.125686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ea3164-b777-4c39-82aa-3aade451f221 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:14.127141 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ea3164-b777-4c39-82aa-3aade451f221 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:14.135324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1144/2277] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:21:14 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:21:14.452138 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee51bb35-4bc4-4eba-99b8-511650cb0cb5 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:14.452420 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee51bb35-4bc4-4eba-99b8-511650cb0cb5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:14.452689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee51bb35-4bc4-4eba-99b8-511650cb0cb5 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:14.454083 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1134/2278] 10.222.0.15 () {54 vars in 921 bytes} [Tue Sep 13 01:21:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:21:14.459389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e56028-ace0-4929-9477-acae2aa4f1de None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:14.459678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e56028-ace0-4929-9477-acae2aa4f1de None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:14.459954 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e56028-ace0-4929-9477-acae2aa4f1de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:14.482122 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-32e56028-ace0-4929-9477-acae2aa4f1de None None] MFA Rules not processed for user `2f2da4d2d3604d16b58dc25b8c17c49a`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:21:14.483256 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-32e56028-ace0-4929-9477-acae2aa4f1de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:14.492291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1145/2279] 10.222.0.15 () {58 vars in 1018 bytes} [Tue Sep 13 01:21:14 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:21:14.503629 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee51bb35-4bc4-4eba-99b8-511650cb0cb5 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:14.505317 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ee51bb35-4bc4-4eba-99b8-511650cb0cb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:14.521087 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee51bb35-4bc4-4eba-99b8-511650cb0cb5 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:14.524809 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39360705-14bc-4823-b0ab-d3be6fd104a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:14.525418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39360705-14bc-4823-b0ab-d3be6fd104a1 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:14.525709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39360705-14bc-4823-b0ab-d3be6fd104a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:14.526019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39360705-14bc-4823-b0ab-d3be6fd104a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:14.554752 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39360705-14bc-4823-b0ab-d3be6fd104a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:14.556816 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39360705-14bc-4823-b0ab-d3be6fd104a1 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:14.568195 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1135/2280] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:21:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:21:14.920899 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e56028-ace0-4929-9477-acae2aa4f1de None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:14.935793 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e56028-ace0-4929-9477-acae2aa4f1de None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:14.941035 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe05e14-91c6-4df6-9281-279756ffc51c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:14.941854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe05e14-91c6-4df6-9281-279756ffc51c service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:14.942139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe05e14-91c6-4df6-9281-279756ffc51c service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:14.942478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe05e14-91c6-4df6-9281-279756ffc51c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:14.944517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7fe05e14-91c6-4df6-9281-279756ffc51c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:15.020690 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe05e14-91c6-4df6-9281-279756ffc51c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:15.022244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe05e14-91c6-4df6-9281-279756ffc51c service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:15.038277 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1146/2281] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:21:14 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:21:17.631273 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a56478d-ea46-4d0a-ac49-c5c82572a02b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:17.633011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a56478d-ea46-4d0a-ac49-c5c82572a02b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:17.633011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a56478d-ea46-4d0a-ac49-c5c82572a02b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:17.652602 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9a56478d-ea46-4d0a-ac49-c5c82572a02b None None] MFA Rules not processed for user `e28a0ba45389417ba8ae92376dba87ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:21:17.655171 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9a56478d-ea46-4d0a-ac49-c5c82572a02b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:17.712850 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1136/2282] 10.222.0.15 () {58 vars in 1009 bytes} [Tue Sep 13 01:21:17 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:21:17.725144 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe05e14-91c6-4df6-9281-279756ffc51c service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:17.726638 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7fe05e14-91c6-4df6-9281-279756ffc51c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:17.767385 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:21:17.806264 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe05e14-91c6-4df6-9281-279756ffc51c service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:17.812851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a2817e-ae15-4313-9357-354e9c6534d5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d7d8ed4ede4ddca192fce0a497689d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:17.813777 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a2817e-ae15-4313-9357-354e9c6534d5 service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:17.814198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a2817e-ae15-4313-9357-354e9c6534d5 service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:17.814616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a2817e-ae15-4313-9357-354e9c6534d5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:17.828312 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a2817e-ae15-4313-9357-354e9c6534d5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:17.830767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a2817e-ae15-4313-9357-354e9c6534d5 service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:17.843367 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1147/2283] 10.222.0.15 () {60 vars in 1384 bytes} [Tue Sep 13 01:21:17 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 01:21:27.706089 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=46 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:21:27.838221 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=16 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:21:39.632411 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ea22e-59f8-4fca-8805-614577be0a2f None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:39.632921 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ea22e-59f8-4fca-8805-614577be0a2f None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:39.633033 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ea22e-59f8-4fca-8805-614577be0a2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:39.651875 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d49ea22e-59f8-4fca-8805-614577be0a2f None None] MFA Rules not processed for user `776733af5b6a4115b2a11d67666a379d`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:21:39.653607 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d49ea22e-59f8-4fca-8805-614577be0a2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:39.705853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1137/2284] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:21:39 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:21:42.314377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a2817e-ae15-4313-9357-354e9c6534d5 service swift] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:42.316683 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-01a2817e-ae15-4313-9357-354e9c6534d5 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:42.403712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a2817e-ae15-4313-9357-354e9c6534d5 service swift] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:42.412362 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f229b8b6-b243-4025-9636-20ecbf627652 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8eaff1b843a145a49994dcf089f44315', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:42.413216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f229b8b6-b243-4025-9636-20ecbf627652 service placement] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:42.413561 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f229b8b6-b243-4025-9636-20ecbf627652 service placement] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:42.413907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f229b8b6-b243-4025-9636-20ecbf627652 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:42.415861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f229b8b6-b243-4025-9636-20ecbf627652 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:42.431177 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f229b8b6-b243-4025-9636-20ecbf627652 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:42.432897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f229b8b6-b243-4025-9636-20ecbf627652 service placement] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:42.444483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1148/2285] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:21:42 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:21:44.007605 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6db70b-18d5-4b1f-943c-344381f7dce0 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:44.008048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6db70b-18d5-4b1f-943c-344381f7dce0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:44.008156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6db70b-18d5-4b1f-943c-344381f7dce0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:44.033517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ef6db70b-18d5-4b1f-943c-344381f7dce0 None None] MFA Rules not processed for user `e28a0ba45389417ba8ae92376dba87ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:21:44.034840 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ef6db70b-18d5-4b1f-943c-344381f7dce0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:21:44.042223 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1138/2286] 10.222.0.15 () {58 vars in 967 bytes} [Tue Sep 13 01:21:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:21:44.055739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f229b8b6-b243-4025-9636-20ecbf627652 service placement] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:21:44.068803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f229b8b6-b243-4025-9636-20ecbf627652 service placement] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:21:44.073730 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2622ba-c417-4582-a893-7d1982dfb266 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d7d8ed4ede4ddca192fce0a497689d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:21:44.074335 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2622ba-c417-4582-a893-7d1982dfb266 service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:21:44.074661 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2622ba-c417-4582-a893-7d1982dfb266 service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:21:44.074966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2622ba-c417-4582-a893-7d1982dfb266 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:21:44.083838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2622ba-c417-4582-a893-7d1982dfb266 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:21:44.085487 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2622ba-c417-4582-a893-7d1982dfb266 service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:21:44.095302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1149/2287] 10.222.0.15 () {60 vars in 1384 bytes} [Tue Sep 13 01:21:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 01:21:54.027613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=28 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:21:54.092040 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=38 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:22:23.986998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa74e49-5c3c-4876-8bf7-3eab7e3f0f57 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:22:23.987540 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa74e49-5c3c-4876-8bf7-3eab7e3f0f57 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:22:23.987540 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa74e49-5c3c-4876-8bf7-3eab7e3f0f57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:22:24.006226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6aa74e49-5c3c-4876-8bf7-3eab7e3f0f57 None None] MFA Rules not processed for user `0c5dcfa5794843a5896fa051a4554e1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:22:24.007362 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6aa74e49-5c3c-4876-8bf7-3eab7e3f0f57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:22:24.061818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1139/2288] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:22:23 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:22:34.054704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=25 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:22:40.806870 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19142254-dbf4-4f29-8758-a1d0981aee8a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:22:40.807368 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19142254-dbf4-4f29-8758-a1d0981aee8a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:22:40.807466 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19142254-dbf4-4f29-8758-a1d0981aee8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:22:40.827256 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-19142254-dbf4-4f29-8758-a1d0981aee8a None None] MFA Rules not processed for user `e2846ad486314958bb74af71b647c470`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:22:40.828445 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-19142254-dbf4-4f29-8758-a1d0981aee8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:22:40.882394 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1150/2289] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:22:40 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:22:50.875728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=25 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:22:57.749788 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fca5eae-5c7b-4615-9fe7-539f6914b9ab None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:22:57.750289 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fca5eae-5c7b-4615-9fe7-539f6914b9ab None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:22:57.750406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fca5eae-5c7b-4615-9fe7-539f6914b9ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:22:57.770474 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9fca5eae-5c7b-4615-9fe7-539f6914b9ab None None] MFA Rules not processed for user `776733af5b6a4115b2a11d67666a379d`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:22:57.771712 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9fca5eae-5c7b-4615-9fe7-539f6914b9ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:22:57.778618 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1140/2290] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:22:57 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:23:07.765053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:17.657298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87208faa-da9b-4caa-a89a-3972a1dabec0 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:17.657767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87208faa-da9b-4caa-a89a-3972a1dabec0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:17.657927 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87208faa-da9b-4caa-a89a-3972a1dabec0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:17.678967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-87208faa-da9b-4caa-a89a-3972a1dabec0 None None] MFA Rules not processed for user `349e19300f1e48c3ac44fc34611c7f4a`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:17.680401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-87208faa-da9b-4caa-a89a-3972a1dabec0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:17.742148 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1151/2291] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:17 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:23:27.732313 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=25 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:31.461387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cb5809-236c-4719-9528-8317bfc22e66 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:31.462255 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cb5809-236c-4719-9528-8317bfc22e66 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:31.462587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cb5809-236c-4719-9528-8317bfc22e66 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:31.463987 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1141/2292] 10.222.0.15 () {54 vars in 921 bytes} [Tue Sep 13 01:23:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:23:31.470619 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9f30df-0ff8-49cb-9b26-a820f9116657 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:31.471472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9f30df-0ff8-49cb-9b26-a820f9116657 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:31.471761 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9f30df-0ff8-49cb-9b26-a820f9116657 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:31.495363 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-de9f30df-0ff8-49cb-9b26-a820f9116657 None None] MFA Rules not processed for user `09cd263f72484733adaaa0850ebfa609`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:31.497290 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-de9f30df-0ff8-49cb-9b26-a820f9116657 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:31.503779 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1152/2293] 10.222.0.15 () {58 vars in 1018 bytes} [Tue Sep 13 01:23:31 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:23:31.518107 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cb5809-236c-4719-9528-8317bfc22e66 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:31.530359 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cb5809-236c-4719-9528-8317bfc22e66 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:31.534243 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f987201-6d0c-4d9c-9760-451cb001c08c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:31.534882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f987201-6d0c-4d9c-9760-451cb001c08c service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:31.535256 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f987201-6d0c-4d9c-9760-451cb001c08c service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:31.536085 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f987201-6d0c-4d9c-9760-451cb001c08c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:31.545425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f987201-6d0c-4d9c-9760-451cb001c08c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:31.547842 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f987201-6d0c-4d9c-9760-451cb001c08c service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:31.558679 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1142/2294] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:23:31 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:23:31.648172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b28294-7dcb-4ba5-a98c-bb606d91e670 None None] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:31.648466 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b28294-7dcb-4ba5-a98c-bb606d91e670 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:31.648746 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b28294-7dcb-4ba5-a98c-bb606d91e670 None None] PATH_INFO: `/` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:31.650272 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1153/2295] 10.222.0.15 () {54 vars in 921 bytes} [Tue Sep 13 01:23:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:23:31.656998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bea313-af06-4e8e-9d5c-2dd429d08b3b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:31.657424 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bea313-af06-4e8e-9d5c-2dd429d08b3b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:31.657790 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bea313-af06-4e8e-9d5c-2dd429d08b3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:31.682193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-84bea313-af06-4e8e-9d5c-2dd429d08b3b None None] MFA Rules not processed for user `2f2da4d2d3604d16b58dc25b8c17c49a`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:31.684172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-84bea313-af06-4e8e-9d5c-2dd429d08b3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:31.700615 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1143/2296] 10.222.0.15 () {58 vars in 1018 bytes} [Tue Sep 13 01:23:31 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:23:31.715103 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b28294-7dcb-4ba5-a98c-bb606d91e670 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:31.729938 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b28294-7dcb-4ba5-a98c-bb606d91e670 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:31.735785 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c53589-a5f2-4409-b2da-746bbbe9eb6a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:31.736692 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c53589-a5f2-4409-b2da-746bbbe9eb6a service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:31.737202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c53589-a5f2-4409-b2da-746bbbe9eb6a service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:31.737693 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c53589-a5f2-4409-b2da-746bbbe9eb6a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:31.750955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c53589-a5f2-4409-b2da-746bbbe9eb6a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:31.753661 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c53589-a5f2-4409-b2da-746bbbe9eb6a service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:31.771293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1154/2297] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:23:31 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:23:32.296544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bea313-af06-4e8e-9d5c-2dd429d08b3b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:32.306040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bea313-af06-4e8e-9d5c-2dd429d08b3b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:32.310549 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f3886e-7267-408f-9df2-2e75e9faecdf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:32.311204 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f3886e-7267-408f-9df2-2e75e9faecdf service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:32.311469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f3886e-7267-408f-9df2-2e75e9faecdf service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:32.311739 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f3886e-7267-408f-9df2-2e75e9faecdf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:32.313690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-95f3886e-7267-408f-9df2-2e75e9faecdf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:32.389479 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f3886e-7267-408f-9df2-2e75e9faecdf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:32.390963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f3886e-7267-408f-9df2-2e75e9faecdf service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:32.418709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1144/2298] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:23:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:23:34.554220 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c53589-a5f2-4409-b2da-746bbbe9eb6a service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:34.556565 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-41c53589-a5f2-4409-b2da-746bbbe9eb6a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:34.574718 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c53589-a5f2-4409-b2da-746bbbe9eb6a service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:34.579391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41f348a-bcd1-4a13-b45c-d975bc4211cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:34.580042 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41f348a-bcd1-4a13-b45c-d975bc4211cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:34.580318 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41f348a-bcd1-4a13-b45c-d975bc4211cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:34.580620 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41f348a-bcd1-4a13-b45c-d975bc4211cc admin admin] PATH_INFO: `/v3/users/0c5dcfa5794843a5896fa051a4554e1f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:34.581846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41f348a-bcd1-4a13-b45c-d975bc4211cc admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c5dcfa5794843a5896fa051a4554e1f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:34.583796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41f348a-bcd1-4a13-b45c-d975bc4211cc admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:34.637007 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f41f348a-bcd1-4a13-b45c-d975bc4211cc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c5dcfa5794843a5896fa051a4554e1f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:34.637755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f41f348a-bcd1-4a13-b45c-d975bc4211cc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c5dcfa5794843a5896fa051a4554e1f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:34.644891 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f41f348a-bcd1-4a13-b45c-d975bc4211cc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c5dcfa5794843a5896fa051a4554e1f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:34.645518 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f41f348a-bcd1-4a13-b45c-d975bc4211cc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c5dcfa5794843a5896fa051a4554e1f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:34.657852 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f41f348a-bcd1-4a13-b45c-d975bc4211cc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c5dcfa5794843a5896fa051a4554e1f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:34.670305 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1155/2299] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:23:34 2022] DELETE /identity/v3/users/0c5dcfa5794843a5896fa051a4554e1f => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:34.846198 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f3886e-7267-408f-9df2-2e75e9faecdf service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:34.856807 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f3886e-7267-408f-9df2-2e75e9faecdf service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:34.917809 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69053746-0722-4aae-a263-a466c9c56f2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:34.918412 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69053746-0722-4aae-a263-a466c9c56f2f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:34.918698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69053746-0722-4aae-a263-a466c9c56f2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:34.919011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69053746-0722-4aae-a263-a466c9c56f2f admin admin] PATH_INFO: `/v3/projects/a84dd36fe9484f41bc2f0d7da4e47c13` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:34.920969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69053746-0722-4aae-a263-a466c9c56f2f admin admin] RBAC: Authorizing `identity:delete_project(project_id=a84dd36fe9484f41bc2f0d7da4e47c13)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:34.923420 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69053746-0722-4aae-a263-a466c9c56f2f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:34.974218 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-69053746-0722-4aae-a263-a466c9c56f2f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a84dd36fe9484f41bc2f0d7da4e47c13', 'request_id': 'req-69053746-0722-4aae-a263-a466c9c56f2f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:34.980390 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-69053746-0722-4aae-a263-a466c9c56f2f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a84dd36fe9484f41bc2f0d7da4e47c13', 'request_id': 'req-69053746-0722-4aae-a263-a466c9c56f2f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:34.986675 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-69053746-0722-4aae-a263-a466c9c56f2f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a84dd36fe9484f41bc2f0d7da4e47c13', 'request_id': 'req-69053746-0722-4aae-a263-a466c9c56f2f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:34.999867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-69053746-0722-4aae-a263-a466c9c56f2f admin admin] The token cache is being invalidate because project a84dd36fe9484f41bc2f0d7da4e47c13 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:23:35.000152 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-69053746-0722-4aae-a263-a466c9c56f2f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:35.000799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-69053746-0722-4aae-a263-a466c9c56f2f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:35.002250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1145/2300] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:23:34 2022] DELETE /identity/v3/projects/a84dd36fe9484f41bc2f0d7da4e47c13 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:35.007173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fd7504-441f-49cc-b427-2cad328b8679 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:35.007487 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fd7504-441f-49cc-b427-2cad328b8679 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:35.007788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fd7504-441f-49cc-b427-2cad328b8679 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:35.027837 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b3fd7504-441f-49cc-b427-2cad328b8679 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:35.029188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b3fd7504-441f-49cc-b427-2cad328b8679 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:35.090378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1156/2301] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:23:35.097984 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4ad4fa-ecff-4e8a-aca6-c00b14cd5793 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:35.098265 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4ad4fa-ecff-4e8a-aca6-c00b14cd5793 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:35.098537 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4ad4fa-ecff-4e8a-aca6-c00b14cd5793 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:35.107533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:35.119107 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-4b4ad4fa-ecff-4e8a-aca6-c00b14cd5793 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:35.120294 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4b4ad4fa-ecff-4e8a-aca6-c00b14cd5793 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:35.128567 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1146/2302] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:23:35.132969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fd7504-441f-49cc-b427-2cad328b8679 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:35.142644 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fd7504-441f-49cc-b427-2cad328b8679 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:35.147392 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ca9a27-f953-476b-8d07-67bff0574bf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:35.148016 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ca9a27-f953-476b-8d07-67bff0574bf6 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:35.148298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ca9a27-f953-476b-8d07-67bff0574bf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:35.148590 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ca9a27-f953-476b-8d07-67bff0574bf6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:35.149472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ca9a27-f953-476b-8d07-67bff0574bf6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:23:35.149816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ca9a27-f953-476b-8d07-67bff0574bf6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:35.152072 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ca9a27-f953-476b-8d07-67bff0574bf6 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:35.163803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1157/2303] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:23:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:23:35.168626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4ad4fa-ecff-4e8a-aca6-c00b14cd5793 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:35.177687 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4ad4fa-ecff-4e8a-aca6-c00b14cd5793 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:35.182790 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d5d442-b468-4c2c-97f9-e9e2cb5a1bab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:35.183997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d5d442-b468-4c2c-97f9-e9e2cb5a1bab admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:35.184258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d5d442-b468-4c2c-97f9-e9e2cb5a1bab admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:35.184529 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d5d442-b468-4c2c-97f9-e9e2cb5a1bab admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:35.185160 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d5d442-b468-4c2c-97f9-e9e2cb5a1bab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:35.187096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d5d442-b468-4c2c-97f9-e9e2cb5a1bab admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:35.215623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1147/2304] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:23:35 2022] POST /identity/v3/projects => generated 405 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:23:35.219930 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ca9a27-f953-476b-8d07-67bff0574bf6 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:35.230494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ca9a27-f953-476b-8d07-67bff0574bf6 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:35.270392 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:35.288589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1ba72f-1ba8-4ddd-b690-910ec1d99b3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:35.289204 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1ba72f-1ba8-4ddd-b690-910ec1d99b3e admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:35.289486 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1ba72f-1ba8-4ddd-b690-910ec1d99b3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:35.289762 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1ba72f-1ba8-4ddd-b690-910ec1d99b3e admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:35.290400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1ba72f-1ba8-4ddd-b690-910ec1d99b3e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:35.292398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1ba72f-1ba8-4ddd-b690-910ec1d99b3e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:35.314592 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1158/2305] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:23:35 2022] POST /identity/v3/users => generated 303 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:23:35.318771 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d5d442-b468-4c2c-97f9-e9e2cb5a1bab admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:35.328843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d5d442-b468-4c2c-97f9-e9e2cb5a1bab admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:35.333766 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feef74b2-bb85-4c4b-af53-9fc8255763eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:35.334384 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feef74b2-bb85-4c4b-af53-9fc8255763eb admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:35.334793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feef74b2-bb85-4c4b-af53-9fc8255763eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:35.335221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feef74b2-bb85-4c4b-af53-9fc8255763eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:35.335877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feef74b2-bb85-4c4b-af53-9fc8255763eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:35.337490 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feef74b2-bb85-4c4b-af53-9fc8255763eb admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:35.348444 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1148/2306] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:23:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:23:35.352690 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1ba72f-1ba8-4ddd-b690-910ec1d99b3e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:35.353730 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feef74b2-bb85-4c4b-af53-9fc8255763eb admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:35.356063 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-feef74b2-bb85-4c4b-af53-9fc8255763eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:35.363305 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1ba72f-1ba8-4ddd-b690-910ec1d99b3e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:35.367884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728fe4f5-0919-42dd-af62-6ccc1413dc3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:35.368482 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728fe4f5-0919-42dd-af62-6ccc1413dc3d admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:35.368756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728fe4f5-0919-42dd-af62-6ccc1413dc3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:35.369023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728fe4f5-0919-42dd-af62-6ccc1413dc3d admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:35.369656 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728fe4f5-0919-42dd-af62-6ccc1413dc3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:35.371366 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728fe4f5-0919-42dd-af62-6ccc1413dc3d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:35.382339 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1159/2307] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:23:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:23:35.386483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728fe4f5-0919-42dd-af62-6ccc1413dc3d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:35.396355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728fe4f5-0919-42dd-af62-6ccc1413dc3d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:35.400984 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd26e63b-466b-41b7-b89c-b0c2b32dd7b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:35.402180 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd26e63b-466b-41b7-b89c-b0c2b32dd7b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:35.402470 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd26e63b-466b-41b7-b89c-b0c2b32dd7b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:35.402741 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd26e63b-466b-41b7-b89c-b0c2b32dd7b0 admin admin] PATH_INFO: `/v3/projects/6bc0fde80c824aa3a52f3007262d5cab/users/ab841b33e00643379c74cf1d550c346b/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:35.422035 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd26e63b-466b-41b7-b89c-b0c2b32dd7b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6bc0fde80c824aa3a52f3007262d5cab, user_id=ab841b33e00643379c74cf1d550c346b, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:35.425441 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd26e63b-466b-41b7-b89c-b0c2b32dd7b0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:35.426054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feef74b2-bb85-4c4b-af53-9fc8255763eb admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:35.429901 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e68094-2682-4c2e-aa9c-aeb7b8cd280a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:35.430548 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e68094-2682-4c2e-aa9c-aeb7b8cd280a service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:35.430825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e68094-2682-4c2e-aa9c-aeb7b8cd280a service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:35.431179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e68094-2682-4c2e-aa9c-aeb7b8cd280a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:35.433207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c3e68094-2682-4c2e-aa9c-aeb7b8cd280a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:35.439796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1160/2308] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:23:35 2022] PUT /identity/v3/projects/6bc0fde80c824aa3a52f3007262d5cab/users/ab841b33e00643379c74cf1d550c346b/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:23:35.444433 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee805cdb-1296-42df-b80a-d5a0ece92c09 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:35.444786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee805cdb-1296-42df-b80a-d5a0ece92c09 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:35.445069 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee805cdb-1296-42df-b80a-d5a0ece92c09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:35.463998 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ee805cdb-1296-42df-b80a-d5a0ece92c09 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:35.465683 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ee805cdb-1296-42df-b80a-d5a0ece92c09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:35.506599 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e68094-2682-4c2e-aa9c-aeb7b8cd280a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:35.508095 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e68094-2682-4c2e-aa9c-aeb7b8cd280a service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:35.519419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1149/2309] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:23:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:23:35.527437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1161/2310] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:23:35.539479 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e68094-2682-4c2e-aa9c-aeb7b8cd280a service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:35.540985 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c3e68094-2682-4c2e-aa9c-aeb7b8cd280a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:35.552584 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e68094-2682-4c2e-aa9c-aeb7b8cd280a service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:35.607050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058938c7-1b1f-4382-9430-ad06da393300 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:35.607675 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058938c7-1b1f-4382-9430-ad06da393300 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:35.607959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058938c7-1b1f-4382-9430-ad06da393300 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:35.608226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058938c7-1b1f-4382-9430-ad06da393300 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:35.618040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-058938c7-1b1f-4382-9430-ad06da393300 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:35.619561 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-058938c7-1b1f-4382-9430-ad06da393300 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:35.634020 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1150/2311] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:23:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:23:36.350515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db22e6f4-b404-4092-8525-963b54e99531 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:36.351050 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db22e6f4-b404-4092-8525-963b54e99531 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:36.351195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db22e6f4-b404-4092-8525-963b54e99531 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:36.382145 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-db22e6f4-b404-4092-8525-963b54e99531 None None] MFA Rules not processed for user `ab841b33e00643379c74cf1d550c346b`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:36.383391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-db22e6f4-b404-4092-8525-963b54e99531 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:36.386868 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058938c7-1b1f-4382-9430-ad06da393300 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:36.396178 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058938c7-1b1f-4382-9430-ad06da393300 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:36.401338 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9de19e3-48ce-4125-a16d-b436996fb918 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:36.403509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9de19e3-48ce-4125-a16d-b436996fb918 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:36.403509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9de19e3-48ce-4125-a16d-b436996fb918 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:36.403509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9de19e3-48ce-4125-a16d-b436996fb918 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:36.405316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b9de19e3-48ce-4125-a16d-b436996fb918 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:36.413719 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:36.480264 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1162/2312] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:36 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:23:36.481168 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db22e6f4-b404-4092-8525-963b54e99531 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:36.483257 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9de19e3-48ce-4125-a16d-b436996fb918 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:36.484705 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9de19e3-48ce-4125-a16d-b436996fb918 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:36.492313 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db22e6f4-b404-4092-8525-963b54e99531 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:36.496847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1151/2313] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:23:36 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:23:36.498158 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69175d90-0521-4908-80d2-29022c61e8b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:36.498906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69175d90-0521-4908-80d2-29022c61e8b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:36.499205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69175d90-0521-4908-80d2-29022c61e8b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:36.499567 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69175d90-0521-4908-80d2-29022c61e8b0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:36.500389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69175d90-0521-4908-80d2-29022c61e8b0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:36.502335 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69175d90-0521-4908-80d2-29022c61e8b0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:36.537323 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1163/2314] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:23:36 2022] POST /identity/v3/projects => generated 403 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:23:36.541300 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9de19e3-48ce-4125-a16d-b436996fb918 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:36.552853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9de19e3-48ce-4125-a16d-b436996fb918 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:36.623416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e7aa85-9a4a-4506-92b3-c4b57c0433cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:36.624040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e7aa85-9a4a-4506-92b3-c4b57c0433cd admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:36.624383 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e7aa85-9a4a-4506-92b3-c4b57c0433cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:36.624659 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e7aa85-9a4a-4506-92b3-c4b57c0433cd admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:36.625297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e7aa85-9a4a-4506-92b3-c4b57c0433cd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:36.627239 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e7aa85-9a4a-4506-92b3-c4b57c0433cd admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:36.656791 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1152/2315] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:23:36 2022] POST /identity/v3/users => generated 308 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:23:36.658365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69175d90-0521-4908-80d2-29022c61e8b0 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:36.664166 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:36.670032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69175d90-0521-4908-80d2-29022c61e8b0 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:36.676520 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c7d968-651e-4c4f-8462-a2f4446a30b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:36.677220 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c7d968-651e-4c4f-8462-a2f4446a30b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:36.677592 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c7d968-651e-4c4f-8462-a2f4446a30b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:36.677968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c7d968-651e-4c4f-8462-a2f4446a30b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:36.678698 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c7d968-651e-4c4f-8462-a2f4446a30b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:36.680391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c7d968-651e-4c4f-8462-a2f4446a30b5 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:36.691168 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1164/2316] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:23:36 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:23:36.695660 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e7aa85-9a4a-4506-92b3-c4b57c0433cd admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:36.706750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e7aa85-9a4a-4506-92b3-c4b57c0433cd admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:36.712162 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9559f66-4b2e-41fb-b7d1-48ee35f4d8ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:36.712960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9559f66-4b2e-41fb-b7d1-48ee35f4d8ce admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:36.713086 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9559f66-4b2e-41fb-b7d1-48ee35f4d8ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:36.713443 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9559f66-4b2e-41fb-b7d1-48ee35f4d8ce admin admin] PATH_INFO: `/v3/projects/9a2110e47a73486aabcacd63cf1adedc/users/dc5e248d35cc41dfbf557c659aa855eb/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:36.730420 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9559f66-4b2e-41fb-b7d1-48ee35f4d8ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a2110e47a73486aabcacd63cf1adedc, user_id=dc5e248d35cc41dfbf557c659aa855eb, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:36.733047 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9559f66-4b2e-41fb-b7d1-48ee35f4d8ce admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:36.747421 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1153/2317] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:23:36 2022] PUT /identity/v3/projects/9a2110e47a73486aabcacd63cf1adedc/users/dc5e248d35cc41dfbf557c659aa855eb/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:36.751389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c7d968-651e-4c4f-8462-a2f4446a30b5 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:36.761985 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c7d968-651e-4c4f-8462-a2f4446a30b5 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:36.821521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826b7f4f-57fd-433e-94ce-5d2af2a43531 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:36.822136 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826b7f4f-57fd-433e-94ce-5d2af2a43531 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:36.822445 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826b7f4f-57fd-433e-94ce-5d2af2a43531 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:36.822713 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826b7f4f-57fd-433e-94ce-5d2af2a43531 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:36.823421 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826b7f4f-57fd-433e-94ce-5d2af2a43531 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:36.825528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826b7f4f-57fd-433e-94ce-5d2af2a43531 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:36.836448 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1165/2318] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:23:36 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:23:36.841106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9559f66-4b2e-41fb-b7d1-48ee35f4d8ce admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:36.851889 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9559f66-4b2e-41fb-b7d1-48ee35f4d8ce admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:36.856532 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcf90ff-6a35-4c13-a5ca-87625f7dcf96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:36.857186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcf90ff-6a35-4c13-a5ca-87625f7dcf96 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:36.857462 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcf90ff-6a35-4c13-a5ca-87625f7dcf96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:36.857733 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcf90ff-6a35-4c13-a5ca-87625f7dcf96 admin admin] PATH_INFO: `/v3/projects/9a2110e47a73486aabcacd63cf1adedc/users/dc5e248d35cc41dfbf557c659aa855eb/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:36.859636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abcf90ff-6a35-4c13-a5ca-87625f7dcf96 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a2110e47a73486aabcacd63cf1adedc, user_id=dc5e248d35cc41dfbf557c659aa855eb, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:36.862246 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abcf90ff-6a35-4c13-a5ca-87625f7dcf96 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:36.874135 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1154/2319] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:23:36 2022] PUT /identity/v3/projects/9a2110e47a73486aabcacd63cf1adedc/users/dc5e248d35cc41dfbf557c659aa855eb/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:23:37.537959 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9a8887-4213-477e-b7f3-864c36d7c713 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:37.538514 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9a8887-4213-477e-b7f3-864c36d7c713 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:37.538747 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9a8887-4213-477e-b7f3-864c36d7c713 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:37.570213 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7d9a8887-4213-477e-b7f3-864c36d7c713 None None] MFA Rules not processed for user `dc5e248d35cc41dfbf557c659aa855eb`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:37.571503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7d9a8887-4213-477e-b7f3-864c36d7c713 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:37.674189 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1166/2320] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:37 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:23:39.131778 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9287b44-6e1d-4c3e-a8f7-40e309e9d895 None None] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:39.132307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9287b44-6e1d-4c3e-a8f7-40e309e9d895 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:39.132424 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9287b44-6e1d-4c3e-a8f7-40e309e9d895 None None] PATH_INFO: `/` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:39.133675 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1155/2321] 10.222.0.15 () {54 vars in 921 bytes} [Tue Sep 13 01:23:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 01:23:39.163534 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a601a63-2827-4fb8-bb6d-3709fc5ca9d0 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:39.164048 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a601a63-2827-4fb8-bb6d-3709fc5ca9d0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:39.164142 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a601a63-2827-4fb8-bb6d-3709fc5ca9d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:39.184851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-1a601a63-2827-4fb8-bb6d-3709fc5ca9d0 None None] MFA Rules not processed for user `8eaff1b843a145a49994dcf089f44315`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:39.186125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1a601a63-2827-4fb8-bb6d-3709fc5ca9d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:39.196243 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9287b44-6e1d-4c3e-a8f7-40e309e9d895 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:39.249976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:39.256360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1167/2322] 10.222.0.15 () {58 vars in 1018 bytes} [Tue Sep 13 01:23:39 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:23:39.267714 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a601a63-2827-4fb8-bb6d-3709fc5ca9d0 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:39.269211 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9287b44-6e1d-4c3e-a8f7-40e309e9d895 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:39.269320 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1a601a63-2827-4fb8-bb6d-3709fc5ca9d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:39.273745 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b749f87e-8fd3-47f7-a233-363add452d73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:39.274435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b749f87e-8fd3-47f7-a233-363add452d73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:39.274720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b749f87e-8fd3-47f7-a233-363add452d73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:39.275027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b749f87e-8fd3-47f7-a233-363add452d73 admin admin] PATH_INFO: `/v3/users/ab841b33e00643379c74cf1d550c346b` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:39.276157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b749f87e-8fd3-47f7-a233-363add452d73 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ab841b33e00643379c74cf1d550c346b)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:39.278920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b749f87e-8fd3-47f7-a233-363add452d73 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:39.283547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a601a63-2827-4fb8-bb6d-3709fc5ca9d0 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:39.290454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e170d59b-d7ab-4c66-a336-ea359baeed57 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8eaff1b843a145a49994dcf089f44315', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:39.291105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e170d59b-d7ab-4c66-a336-ea359baeed57 service placement] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:39.291387 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e170d59b-d7ab-4c66-a336-ea359baeed57 service placement] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:39.291664 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e170d59b-d7ab-4c66-a336-ea359baeed57 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:39.299932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:39.302375 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e170d59b-d7ab-4c66-a336-ea359baeed57 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:39.304590 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e170d59b-d7ab-4c66-a336-ea359baeed57 service placement] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:39.319138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1168/2323] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:23:39 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:23:39.336080 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-b749f87e-8fd3-47f7-a233-363add452d73 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ab841b33e00643379c74cf1d550c346b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:39.336829 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-b749f87e-8fd3-47f7-a233-363add452d73 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ab841b33e00643379c74cf1d550c346b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:39.345331 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-b749f87e-8fd3-47f7-a233-363add452d73 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ab841b33e00643379c74cf1d550c346b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:39.346100 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-b749f87e-8fd3-47f7-a233-363add452d73 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ab841b33e00643379c74cf1d550c346b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:39.353366 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-b749f87e-8fd3-47f7-a233-363add452d73 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ab841b33e00643379c74cf1d550c346b'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:39.372401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1156/2324] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:23:39 2022] DELETE /identity/v3/users/ab841b33e00643379c74cf1d550c346b => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:39.594578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e170d59b-d7ab-4c66-a336-ea359baeed57 service placement] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:39.608557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e170d59b-d7ab-4c66-a336-ea359baeed57 service placement] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:39.673816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dd1909-7ff3-4ffd-882f-5285d2af3739 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:39.674455 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dd1909-7ff3-4ffd-882f-5285d2af3739 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:39.674735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dd1909-7ff3-4ffd-882f-5285d2af3739 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:39.675042 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dd1909-7ff3-4ffd-882f-5285d2af3739 admin admin] PATH_INFO: `/v3/projects/6bc0fde80c824aa3a52f3007262d5cab` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:39.676184 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05dd1909-7ff3-4ffd-882f-5285d2af3739 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6bc0fde80c824aa3a52f3007262d5cab)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:39.678134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05dd1909-7ff3-4ffd-882f-5285d2af3739 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:39.727043 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-05dd1909-7ff3-4ffd-882f-5285d2af3739 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6bc0fde80c824aa3a52f3007262d5cab', 'request_id': 'req-05dd1909-7ff3-4ffd-882f-5285d2af3739'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:39.733670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-05dd1909-7ff3-4ffd-882f-5285d2af3739 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6bc0fde80c824aa3a52f3007262d5cab', 'request_id': 'req-05dd1909-7ff3-4ffd-882f-5285d2af3739'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:39.739874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-05dd1909-7ff3-4ffd-882f-5285d2af3739 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6bc0fde80c824aa3a52f3007262d5cab', 'request_id': 'req-05dd1909-7ff3-4ffd-882f-5285d2af3739'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:39.754754 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-05dd1909-7ff3-4ffd-882f-5285d2af3739 admin admin] The token cache is being invalidate because project 6bc0fde80c824aa3a52f3007262d5cab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:23:39.755099 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-05dd1909-7ff3-4ffd-882f-5285d2af3739 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:39.755727 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-05dd1909-7ff3-4ffd-882f-5285d2af3739 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:39.757209 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1169/2325] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:23:39 2022] DELETE /identity/v3/projects/6bc0fde80c824aa3a52f3007262d5cab => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:39.761368 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b749f87e-8fd3-47f7-a233-363add452d73 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:39.762945 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b749f87e-8fd3-47f7-a233-363add452d73 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:39.775908 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b749f87e-8fd3-47f7-a233-363add452d73 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:39.835898 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4bd7c6-c0ea-4899-94cd-dfc5a57f9e0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:39.836446 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4bd7c6-c0ea-4899-94cd-dfc5a57f9e0a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:39.836726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4bd7c6-c0ea-4899-94cd-dfc5a57f9e0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:39.837003 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4bd7c6-c0ea-4899-94cd-dfc5a57f9e0a admin admin] PATH_INFO: `/v3/users/dc5e248d35cc41dfbf557c659aa855eb` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:39.838129 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4bd7c6-c0ea-4899-94cd-dfc5a57f9e0a admin admin] RBAC: Authorizing `identity:delete_user(user_id=dc5e248d35cc41dfbf557c659aa855eb)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:39.840702 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4bd7c6-c0ea-4899-94cd-dfc5a57f9e0a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:39.890831 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4b4bd7c6-c0ea-4899-94cd-dfc5a57f9e0a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dc5e248d35cc41dfbf557c659aa855eb'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:39.891562 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4b4bd7c6-c0ea-4899-94cd-dfc5a57f9e0a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dc5e248d35cc41dfbf557c659aa855eb'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:39.900113 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4b4bd7c6-c0ea-4899-94cd-dfc5a57f9e0a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dc5e248d35cc41dfbf557c659aa855eb'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:39.900715 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4b4bd7c6-c0ea-4899-94cd-dfc5a57f9e0a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dc5e248d35cc41dfbf557c659aa855eb'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:39.913858 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4b4bd7c6-c0ea-4899-94cd-dfc5a57f9e0a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dc5e248d35cc41dfbf557c659aa855eb'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:39.918981 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:39.937075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1157/2326] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:23:39 2022] DELETE /identity/v3/users/dc5e248d35cc41dfbf557c659aa855eb => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:40.125970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dd1909-7ff3-4ffd-882f-5285d2af3739 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:40.127672 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-05dd1909-7ff3-4ffd-882f-5285d2af3739 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:40.145963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dd1909-7ff3-4ffd-882f-5285d2af3739 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:40.236026 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61befc0-0660-4b71-843b-1407a5fe6dc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:40.236857 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61befc0-0660-4b71-843b-1407a5fe6dc0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:40.237139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61befc0-0660-4b71-843b-1407a5fe6dc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:40.237415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61befc0-0660-4b71-843b-1407a5fe6dc0 admin admin] PATH_INFO: `/v3/projects/9a2110e47a73486aabcacd63cf1adedc` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:40.238593 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61befc0-0660-4b71-843b-1407a5fe6dc0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a2110e47a73486aabcacd63cf1adedc)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:40.240607 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61befc0-0660-4b71-843b-1407a5fe6dc0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:40.293736 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:40.335373 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c61befc0-0660-4b71-843b-1407a5fe6dc0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a2110e47a73486aabcacd63cf1adedc', 'request_id': 'req-c61befc0-0660-4b71-843b-1407a5fe6dc0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:40.357153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c61befc0-0660-4b71-843b-1407a5fe6dc0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a2110e47a73486aabcacd63cf1adedc', 'request_id': 'req-c61befc0-0660-4b71-843b-1407a5fe6dc0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:40.366601 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c61befc0-0660-4b71-843b-1407a5fe6dc0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a2110e47a73486aabcacd63cf1adedc', 'request_id': 'req-c61befc0-0660-4b71-843b-1407a5fe6dc0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:40.388117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c61befc0-0660-4b71-843b-1407a5fe6dc0 admin admin] The token cache is being invalidate because project 9a2110e47a73486aabcacd63cf1adedc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:23:40.388444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c61befc0-0660-4b71-843b-1407a5fe6dc0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:40.389306 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c61befc0-0660-4b71-843b-1407a5fe6dc0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:40.391058 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1170/2327] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:23:40 2022] DELETE /identity/v3/projects/9a2110e47a73486aabcacd63cf1adedc => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:40.395710 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8cfa293-3acb-4f3e-9b3e-983c6cb07fed None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:40.396188 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8cfa293-3acb-4f3e-9b3e-983c6cb07fed None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:40.396266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8cfa293-3acb-4f3e-9b3e-983c6cb07fed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:40.419686 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b8cfa293-3acb-4f3e-9b3e-983c6cb07fed None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:40.420920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b8cfa293-3acb-4f3e-9b3e-983c6cb07fed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:40.484460 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1158/2328] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:23:40.492230 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881f1a46-f700-4b10-81fc-1f1f507d1ee1 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:40.492514 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881f1a46-f700-4b10-81fc-1f1f507d1ee1 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:40.492785 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881f1a46-f700-4b10-81fc-1f1f507d1ee1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:40.514356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-881f1a46-f700-4b10-81fc-1f1f507d1ee1 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:40.515592 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-881f1a46-f700-4b10-81fc-1f1f507d1ee1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:40.525860 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1171/2329] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:23:40.530082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8cfa293-3acb-4f3e-9b3e-983c6cb07fed None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:40.540178 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8cfa293-3acb-4f3e-9b3e-983c6cb07fed None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:40.546265 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e1185c-113c-4400-b1f9-9b58e2e56ca9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:40.546929 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e1185c-113c-4400-b1f9-9b58e2e56ca9 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:40.547185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e1185c-113c-4400-b1f9-9b58e2e56ca9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:40.547462 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e1185c-113c-4400-b1f9-9b58e2e56ca9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:40.548086 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e1185c-113c-4400-b1f9-9b58e2e56ca9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:23:40.548388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e1185c-113c-4400-b1f9-9b58e2e56ca9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:40.550524 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e1185c-113c-4400-b1f9-9b58e2e56ca9 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:40.567132 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1159/2330] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:23:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:23:40.568295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881f1a46-f700-4b10-81fc-1f1f507d1ee1 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:40.579042 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881f1a46-f700-4b10-81fc-1f1f507d1ee1 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:40.583781 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bd2758-1c02-4b13-9dce-3cf90a2ebba4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:40.584547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bd2758-1c02-4b13-9dce-3cf90a2ebba4 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:40.584841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bd2758-1c02-4b13-9dce-3cf90a2ebba4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:40.585097 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bd2758-1c02-4b13-9dce-3cf90a2ebba4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:40.585777 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4bd2758-1c02-4b13-9dce-3cf90a2ebba4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:40.587733 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4bd2758-1c02-4b13-9dce-3cf90a2ebba4 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:40.619889 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1172/2331] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:23:40 2022] POST /identity/v3/projects => generated 379 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:23:40.625739 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e1185c-113c-4400-b1f9-9b58e2e56ca9 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:40.638938 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e1185c-113c-4400-b1f9-9b58e2e56ca9 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:40.715017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8944bca-cb39-4f78-98ec-aad865ea3782 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:40.715538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8944bca-cb39-4f78-98ec-aad865ea3782 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:40.715992 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8944bca-cb39-4f78-98ec-aad865ea3782 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:40.716296 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8944bca-cb39-4f78-98ec-aad865ea3782 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:40.717156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8944bca-cb39-4f78-98ec-aad865ea3782 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:40.720130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8944bca-cb39-4f78-98ec-aad865ea3782 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:40.760079 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1160/2332] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:23:40 2022] POST /identity/v3/users => generated 290 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:23:40.761839 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bd2758-1c02-4b13-9dce-3cf90a2ebba4 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:40.773823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bd2758-1c02-4b13-9dce-3cf90a2ebba4 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:40.779077 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76675784-d5f6-4d4f-87cc-bae28447f607 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:40.779734 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76675784-d5f6-4d4f-87cc-bae28447f607 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:40.780012 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76675784-d5f6-4d4f-87cc-bae28447f607 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:40.780302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76675784-d5f6-4d4f-87cc-bae28447f607 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:40.780952 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76675784-d5f6-4d4f-87cc-bae28447f607 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:40.782608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76675784-d5f6-4d4f-87cc-bae28447f607 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:40.793720 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1173/2333] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:23:40 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:23:40.797876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8944bca-cb39-4f78-98ec-aad865ea3782 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:40.809056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8944bca-cb39-4f78-98ec-aad865ea3782 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:40.813802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c66d914-d53f-4f0f-8698-9840a8b82b89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:40.814444 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c66d914-d53f-4f0f-8698-9840a8b82b89 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:40.814726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c66d914-d53f-4f0f-8698-9840a8b82b89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:40.815036 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c66d914-d53f-4f0f-8698-9840a8b82b89 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:40.815676 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c66d914-d53f-4f0f-8698-9840a8b82b89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:40.817316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c66d914-d53f-4f0f-8698-9840a8b82b89 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:40.829988 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1161/2334] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:23:40 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:23:40.834456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76675784-d5f6-4d4f-87cc-bae28447f607 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:40.844145 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76675784-d5f6-4d4f-87cc-bae28447f607 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:40.851179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31658cbb-852f-4114-a486-e52742eee8a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:40.851954 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31658cbb-852f-4114-a486-e52742eee8a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:40.852295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31658cbb-852f-4114-a486-e52742eee8a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:40.852634 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31658cbb-852f-4114-a486-e52742eee8a9 admin admin] PATH_INFO: `/v3/projects/d04e1b05c7ae4270b020ac6bd59a35d9/users/6e55f4a705384e229694c0fb23861683/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:40.872721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31658cbb-852f-4114-a486-e52742eee8a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d04e1b05c7ae4270b020ac6bd59a35d9, user_id=6e55f4a705384e229694c0fb23861683, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:40.875721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31658cbb-852f-4114-a486-e52742eee8a9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:40.901240 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1174/2335] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:23:40 2022] PUT /identity/v3/projects/d04e1b05c7ae4270b020ac6bd59a35d9/users/6e55f4a705384e229694c0fb23861683/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:40.913449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff03a91f-9e7f-4cb0-a567-23ca03cc19d7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:40.913735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff03a91f-9e7f-4cb0-a567-23ca03cc19d7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:40.914175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff03a91f-9e7f-4cb0-a567-23ca03cc19d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:40.953055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ff03a91f-9e7f-4cb0-a567-23ca03cc19d7 None None] MFA Rules not processed for user `6e55f4a705384e229694c0fb23861683`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:40.955039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ff03a91f-9e7f-4cb0-a567-23ca03cc19d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:41.003239 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:DELETE=2,keystone:SELECT=95,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:41.040950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1162/2336] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:40 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:23:41.046005 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31658cbb-852f-4114-a486-e52742eee8a9 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:41.153620 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31658cbb-852f-4114-a486-e52742eee8a9 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:41.158632 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a515ab-d1e9-46c7-84a6-c063e2493fa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:41.159271 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a515ab-d1e9-46c7-84a6-c063e2493fa7 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:41.159553 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a515ab-d1e9-46c7-84a6-c063e2493fa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:41.159829 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a515ab-d1e9-46c7-84a6-c063e2493fa7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:41.160462 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a515ab-d1e9-46c7-84a6-c063e2493fa7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:41.162415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a515ab-d1e9-46c7-84a6-c063e2493fa7 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:41.194040 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1175/2337] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:23:41 2022] POST /identity/v3/projects => generated 381 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:23:41.204332 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff03a91f-9e7f-4cb0-a567-23ca03cc19d7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:41.219249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff03a91f-9e7f-4cb0-a567-23ca03cc19d7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:41.285013 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8538968f-a8c4-4e69-8b67-fa1dbbfbc1aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:41.285639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8538968f-a8c4-4e69-8b67-fa1dbbfbc1aa admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:41.286282 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8538968f-a8c4-4e69-8b67-fa1dbbfbc1aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:41.286282 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8538968f-a8c4-4e69-8b67-fa1dbbfbc1aa admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:41.286814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8538968f-a8c4-4e69-8b67-fa1dbbfbc1aa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:41.288842 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8538968f-a8c4-4e69-8b67-fa1dbbfbc1aa admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:41.312082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1163/2338] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:23:41 2022] POST /identity/v3/users => generated 297 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:23:41.316114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a515ab-d1e9-46c7-84a6-c063e2493fa7 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:41.326435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a515ab-d1e9-46c7-84a6-c063e2493fa7 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:41.332310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8edb530-dc91-4759-89b4-8e5afac7c509 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:41.333610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8edb530-dc91-4759-89b4-8e5afac7c509 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:41.333911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8edb530-dc91-4759-89b4-8e5afac7c509 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:41.334171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8edb530-dc91-4759-89b4-8e5afac7c509 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:41.334815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8edb530-dc91-4759-89b4-8e5afac7c509 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:41.336437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8edb530-dc91-4759-89b4-8e5afac7c509 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:41.349211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1176/2339] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:23:41 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:23:41.353429 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8538968f-a8c4-4e69-8b67-fa1dbbfbc1aa admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:41.365140 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8538968f-a8c4-4e69-8b67-fa1dbbfbc1aa admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:41.371408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4b6a1c-e570-4dc3-9a27-cebafac5af40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:41.371595 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4b6a1c-e570-4dc3-9a27-cebafac5af40 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:41.371811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4b6a1c-e570-4dc3-9a27-cebafac5af40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:41.372065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4b6a1c-e570-4dc3-9a27-cebafac5af40 admin admin] PATH_INFO: `/v3/projects/2a9e0ce2baa24cbcb8861850c21a7776/users/1c3234a3996c4b08b91646fe966d3e6d/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:41.390227 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4b6a1c-e570-4dc3-9a27-cebafac5af40 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a9e0ce2baa24cbcb8861850c21a7776, user_id=1c3234a3996c4b08b91646fe966d3e6d, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:41.394573 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4b6a1c-e570-4dc3-9a27-cebafac5af40 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:41.414917 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1164/2340] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:23:41 2022] PUT /identity/v3/projects/2a9e0ce2baa24cbcb8861850c21a7776/users/1c3234a3996c4b08b91646fe966d3e6d/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:41.418928 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8edb530-dc91-4759-89b4-8e5afac7c509 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:41.429719 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8edb530-dc91-4759-89b4-8e5afac7c509 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:41.453856 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:41.491404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d915ecad-d34a-44c1-9e52-0d696792f1f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:41.492266 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d915ecad-d34a-44c1-9e52-0d696792f1f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:41.492724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d915ecad-d34a-44c1-9e52-0d696792f1f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:41.493142 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d915ecad-d34a-44c1-9e52-0d696792f1f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:41.493931 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d915ecad-d34a-44c1-9e52-0d696792f1f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:41.495690 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d915ecad-d34a-44c1-9e52-0d696792f1f1 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:41.507358 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1177/2341] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:23:41 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:23:41.511249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4b6a1c-e570-4dc3-9a27-cebafac5af40 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:41.521940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4b6a1c-e570-4dc3-9a27-cebafac5af40 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:41.528460 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367e2f02-9195-48b0-a5b6-101bbdaa0fff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:41.529416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367e2f02-9195-48b0-a5b6-101bbdaa0fff admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:41.530270 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367e2f02-9195-48b0-a5b6-101bbdaa0fff admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:41.530746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367e2f02-9195-48b0-a5b6-101bbdaa0fff admin admin] PATH_INFO: `/v3/projects/2a9e0ce2baa24cbcb8861850c21a7776/users/1c3234a3996c4b08b91646fe966d3e6d/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:41.532886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367e2f02-9195-48b0-a5b6-101bbdaa0fff admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a9e0ce2baa24cbcb8861850c21a7776, user_id=1c3234a3996c4b08b91646fe966d3e6d, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:41.536572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367e2f02-9195-48b0-a5b6-101bbdaa0fff admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:41.549393 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1165/2342] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:23:41 2022] PUT /identity/v3/projects/2a9e0ce2baa24cbcb8861850c21a7776/users/1c3234a3996c4b08b91646fe966d3e6d/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:41.562822 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04e12a6-e2da-48de-9f1c-b93b14e93f8b None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:41.563317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04e12a6-e2da-48de-9f1c-b93b14e93f8b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:41.563443 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04e12a6-e2da-48de-9f1c-b93b14e93f8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:41.600399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b04e12a6-e2da-48de-9f1c-b93b14e93f8b None None] MFA Rules not processed for user `1c3234a3996c4b08b91646fe966d3e6d`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:41.602578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b04e12a6-e2da-48de-9f1c-b93b14e93f8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:41.689216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1178/2343] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:41 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:23:41.700951 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367e2f02-9195-48b0-a5b6-101bbdaa0fff admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:41.702729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-367e2f02-9195-48b0-a5b6-101bbdaa0fff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:41.764552 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367e2f02-9195-48b0-a5b6-101bbdaa0fff admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:41.768437 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d063e7e-ec96-4aa6-80c1-9418cf224f48 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:41.769057 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d063e7e-ec96-4aa6-80c1-9418cf224f48 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:41.769339 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d063e7e-ec96-4aa6-80c1-9418cf224f48 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:41.769609 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d063e7e-ec96-4aa6-80c1-9418cf224f48 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:41.779862 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d063e7e-ec96-4aa6-80c1-9418cf224f48 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:41.781376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d063e7e-ec96-4aa6-80c1-9418cf224f48 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:41.793748 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1166/2344] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:23:41 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:23:41.884677 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04e12a6-e2da-48de-9f1c-b93b14e93f8b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:41.896830 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04e12a6-e2da-48de-9f1c-b93b14e93f8b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:41.900843 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfcbe92-7d48-4d31-800a-c35e368adc59 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:41.901731 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfcbe92-7d48-4d31-800a-c35e368adc59 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:41.901854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfcbe92-7d48-4d31-800a-c35e368adc59 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:41.902226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfcbe92-7d48-4d31-800a-c35e368adc59 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:41.988895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfcbe92-7d48-4d31-800a-c35e368adc59 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:41.990317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfcbe92-7d48-4d31-800a-c35e368adc59 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:42.000224 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1179/2345] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:23:41 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:23:44.221467 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0918742-c2e1-42ba-8504-f4e5ed146290 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:44.221935 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0918742-c2e1-42ba-8504-f4e5ed146290 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:44.222196 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0918742-c2e1-42ba-8504-f4e5ed146290 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:44.241348 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-f0918742-c2e1-42ba-8504-f4e5ed146290 None None] MFA Rules not processed for user `2842b222fe6b4821a670d13ec8febf1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:44.242613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f0918742-c2e1-42ba-8504-f4e5ed146290 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:44.272061 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:44.298369 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1167/2346] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:44 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:23:51.108027 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfcbe92-7d48-4d31-800a-c35e368adc59 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:51.157012 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:51.172109 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfcbe92-7d48-4d31-800a-c35e368adc59 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:51.176554 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75cabc51-e6e4-4b9e-8010-7b8426cf73ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:51.177179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cabc51-e6e4-4b9e-8010-7b8426cf73ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:51.177455 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cabc51-e6e4-4b9e-8010-7b8426cf73ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:51.177726 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cabc51-e6e4-4b9e-8010-7b8426cf73ba admin admin] PATH_INFO: `/v3/users/6e55f4a705384e229694c0fb23861683` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:51.178831 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75cabc51-e6e4-4b9e-8010-7b8426cf73ba admin admin] RBAC: Authorizing `identity:delete_user(user_id=6e55f4a705384e229694c0fb23861683)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:51.180755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75cabc51-e6e4-4b9e-8010-7b8426cf73ba admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:51.228855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-75cabc51-e6e4-4b9e-8010-7b8426cf73ba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6e55f4a705384e229694c0fb23861683'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:51.229511 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-75cabc51-e6e4-4b9e-8010-7b8426cf73ba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6e55f4a705384e229694c0fb23861683'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:51.237185 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-75cabc51-e6e4-4b9e-8010-7b8426cf73ba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6e55f4a705384e229694c0fb23861683'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:51.237937 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-75cabc51-e6e4-4b9e-8010-7b8426cf73ba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6e55f4a705384e229694c0fb23861683'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:51.250472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-75cabc51-e6e4-4b9e-8010-7b8426cf73ba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6e55f4a705384e229694c0fb23861683'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:51.263947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1180/2347] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:23:51 2022] DELETE /identity/v3/users/6e55f4a705384e229694c0fb23861683 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:51.268623 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634bc3c0-3889-4a83-8e39-0374da05dc99 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:51.269082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634bc3c0-3889-4a83-8e39-0374da05dc99 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:51.269175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634bc3c0-3889-4a83-8e39-0374da05dc99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:51.288452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-634bc3c0-3889-4a83-8e39-0374da05dc99 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:51.289630 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-634bc3c0-3889-4a83-8e39-0374da05dc99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:51.351955 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1168/2348] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:23:51.361587 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75cabc51-e6e4-4b9e-8010-7b8426cf73ba admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:51.363016 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-75cabc51-e6e4-4b9e-8010-7b8426cf73ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:51.420544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75cabc51-e6e4-4b9e-8010-7b8426cf73ba admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:51.424238 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd096b16-04d1-404d-956c-f1debefc3675 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:51.424839 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd096b16-04d1-404d-956c-f1debefc3675 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:51.425117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd096b16-04d1-404d-956c-f1debefc3675 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:51.425383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd096b16-04d1-404d-956c-f1debefc3675 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:51.434256 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd096b16-04d1-404d-956c-f1debefc3675 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:51.435700 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd096b16-04d1-404d-956c-f1debefc3675 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:51.444909 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1181/2349] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:23:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:23:51.711915 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634bc3c0-3889-4a83-8e39-0374da05dc99 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:51.713352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-634bc3c0-3889-4a83-8e39-0374da05dc99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:51.728276 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634bc3c0-3889-4a83-8e39-0374da05dc99 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:51.734127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec237bb-5158-408e-8b37-c715d0e3fc7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:51.734812 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec237bb-5158-408e-8b37-c715d0e3fc7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:51.735260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec237bb-5158-408e-8b37-c715d0e3fc7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:51.735547 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec237bb-5158-408e-8b37-c715d0e3fc7f admin admin] PATH_INFO: `/v3/projects/d04e1b05c7ae4270b020ac6bd59a35d9` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:51.736652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec237bb-5158-408e-8b37-c715d0e3fc7f admin admin] RBAC: Authorizing `identity:delete_project(project_id=d04e1b05c7ae4270b020ac6bd59a35d9)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:51.738547 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec237bb-5158-408e-8b37-c715d0e3fc7f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:51.785223 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cec237bb-5158-408e-8b37-c715d0e3fc7f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd04e1b05c7ae4270b020ac6bd59a35d9', 'request_id': 'req-cec237bb-5158-408e-8b37-c715d0e3fc7f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:51.790736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cec237bb-5158-408e-8b37-c715d0e3fc7f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd04e1b05c7ae4270b020ac6bd59a35d9', 'request_id': 'req-cec237bb-5158-408e-8b37-c715d0e3fc7f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:51.796949 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cec237bb-5158-408e-8b37-c715d0e3fc7f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd04e1b05c7ae4270b020ac6bd59a35d9', 'request_id': 'req-cec237bb-5158-408e-8b37-c715d0e3fc7f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:51.810799 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cec237bb-5158-408e-8b37-c715d0e3fc7f admin admin] The token cache is being invalidate because project d04e1b05c7ae4270b020ac6bd59a35d9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:23:51.811128 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cec237bb-5158-408e-8b37-c715d0e3fc7f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:51.811746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cec237bb-5158-408e-8b37-c715d0e3fc7f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:51.813205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1169/2350] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:23:51 2022] DELETE /identity/v3/projects/d04e1b05c7ae4270b020ac6bd59a35d9 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:51.817316 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd096b16-04d1-404d-956c-f1debefc3675 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:51.827440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd096b16-04d1-404d-956c-f1debefc3675 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:51.882277 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05dd7a3-693f-48fb-9c59-9f8b58b859ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:51.882931 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05dd7a3-693f-48fb-9c59-9f8b58b859ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:51.883207 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05dd7a3-693f-48fb-9c59-9f8b58b859ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:51.883475 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05dd7a3-693f-48fb-9c59-9f8b58b859ef admin admin] PATH_INFO: `/v3/users/1c3234a3996c4b08b91646fe966d3e6d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:51.884570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d05dd7a3-693f-48fb-9c59-9f8b58b859ef admin admin] RBAC: Authorizing `identity:delete_user(user_id=1c3234a3996c4b08b91646fe966d3e6d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:51.886466 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d05dd7a3-693f-48fb-9c59-9f8b58b859ef admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:51.896828 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:51.935914 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d05dd7a3-693f-48fb-9c59-9f8b58b859ef admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1c3234a3996c4b08b91646fe966d3e6d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:51.936556 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d05dd7a3-693f-48fb-9c59-9f8b58b859ef admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1c3234a3996c4b08b91646fe966d3e6d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:51.946574 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d05dd7a3-693f-48fb-9c59-9f8b58b859ef admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1c3234a3996c4b08b91646fe966d3e6d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:51.947233 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d05dd7a3-693f-48fb-9c59-9f8b58b859ef admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1c3234a3996c4b08b91646fe966d3e6d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:51.959972 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d05dd7a3-693f-48fb-9c59-9f8b58b859ef admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1c3234a3996c4b08b91646fe966d3e6d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:51.972449 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1182/2351] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:23:51 2022] DELETE /identity/v3/users/1c3234a3996c4b08b91646fe966d3e6d => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:23:52.300721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec237bb-5158-408e-8b37-c715d0e3fc7f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:52.302420 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cec237bb-5158-408e-8b37-c715d0e3fc7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:52.318699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec237bb-5158-408e-8b37-c715d0e3fc7f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:52.384202 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a64251-6521-4858-ae49-ce94bf194465 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:52.384839 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a64251-6521-4858-ae49-ce94bf194465 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:52.385146 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a64251-6521-4858-ae49-ce94bf194465 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:52.385419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a64251-6521-4858-ae49-ce94bf194465 admin admin] PATH_INFO: `/v3/projects/2a9e0ce2baa24cbcb8861850c21a7776` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:52.386798 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a64251-6521-4858-ae49-ce94bf194465 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2a9e0ce2baa24cbcb8861850c21a7776)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:52.388720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a64251-6521-4858-ae49-ce94bf194465 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:52.395942 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:52.445362 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d1a64251-6521-4858-ae49-ce94bf194465 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2a9e0ce2baa24cbcb8861850c21a7776', 'request_id': 'req-d1a64251-6521-4858-ae49-ce94bf194465'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:52.450765 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d1a64251-6521-4858-ae49-ce94bf194465 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2a9e0ce2baa24cbcb8861850c21a7776', 'request_id': 'req-d1a64251-6521-4858-ae49-ce94bf194465'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:52.456877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d1a64251-6521-4858-ae49-ce94bf194465 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2a9e0ce2baa24cbcb8861850c21a7776', 'request_id': 'req-d1a64251-6521-4858-ae49-ce94bf194465'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:52.470030 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d1a64251-6521-4858-ae49-ce94bf194465 admin admin] The token cache is being invalidate because project 2a9e0ce2baa24cbcb8861850c21a7776 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:23:52.470338 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d1a64251-6521-4858-ae49-ce94bf194465 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:52.471091 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-d1a64251-6521-4858-ae49-ce94bf194465 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:52.472579 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1170/2352] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:23:52 2022] DELETE /identity/v3/projects/2a9e0ce2baa24cbcb8861850c21a7776 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:52.477569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6982d7-3c0e-4c9b-87e2-e9d6568a2f24 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:52.478079 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6982d7-3c0e-4c9b-87e2-e9d6568a2f24 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:52.478177 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6982d7-3c0e-4c9b-87e2-e9d6568a2f24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:52.497804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-2d6982d7-3c0e-4c9b-87e2-e9d6568a2f24 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:52.499007 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2d6982d7-3c0e-4c9b-87e2-e9d6568a2f24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:52.558259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1183/2353] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:23:52.566073 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443f9362-2340-428b-9697-5505e768e612 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:52.566394 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443f9362-2340-428b-9697-5505e768e612 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:52.566671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443f9362-2340-428b-9697-5505e768e612 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:52.587909 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-443f9362-2340-428b-9697-5505e768e612 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:52.589121 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-443f9362-2340-428b-9697-5505e768e612 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:52.597564 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1171/2354] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:23:52.602483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6982d7-3c0e-4c9b-87e2-e9d6568a2f24 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:52.612786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6982d7-3c0e-4c9b-87e2-e9d6568a2f24 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:52.617307 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed71247-3a6d-4915-a4a6-5f65be6ff2f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:52.617894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed71247-3a6d-4915-a4a6-5f65be6ff2f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:52.618170 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed71247-3a6d-4915-a4a6-5f65be6ff2f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:52.618439 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed71247-3a6d-4915-a4a6-5f65be6ff2f0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:52.619105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed71247-3a6d-4915-a4a6-5f65be6ff2f0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:23:52.619405 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed71247-3a6d-4915-a4a6-5f65be6ff2f0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:52.620969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed71247-3a6d-4915-a4a6-5f65be6ff2f0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:52.632325 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1184/2355] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:23:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:23:52.636595 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443f9362-2340-428b-9697-5505e768e612 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:52.645701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443f9362-2340-428b-9697-5505e768e612 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:52.650359 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4265819-43cf-4dd0-8ea7-da4ae0ddbce4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:52.650987 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4265819-43cf-4dd0-8ea7-da4ae0ddbce4 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:52.651269 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4265819-43cf-4dd0-8ea7-da4ae0ddbce4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:52.651540 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4265819-43cf-4dd0-8ea7-da4ae0ddbce4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:52.652653 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4265819-43cf-4dd0-8ea7-da4ae0ddbce4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:52.654565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4265819-43cf-4dd0-8ea7-da4ae0ddbce4 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:52.681527 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1172/2356] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:23:52 2022] POST /identity/v3/projects => generated 369 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:23:52.685845 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed71247-3a6d-4915-a4a6-5f65be6ff2f0 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:52.695555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed71247-3a6d-4915-a4a6-5f65be6ff2f0 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:52.753456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213e3ded-4a9c-402c-84ce-cb7386e0b8cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:52.754061 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213e3ded-4a9c-402c-84ce-cb7386e0b8cc admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:52.754341 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213e3ded-4a9c-402c-84ce-cb7386e0b8cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:52.754612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213e3ded-4a9c-402c-84ce-cb7386e0b8cc admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:52.755273 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213e3ded-4a9c-402c-84ce-cb7386e0b8cc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:52.757167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213e3ded-4a9c-402c-84ce-cb7386e0b8cc admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:52.783192 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1185/2357] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:23:52 2022] POST /identity/v3/users => generated 285 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:23:52.784361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4265819-43cf-4dd0-8ea7-da4ae0ddbce4 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:52.795394 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4265819-43cf-4dd0-8ea7-da4ae0ddbce4 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:52.800139 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321f1ed4-ec2e-41c8-a2f8-526cac573afa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:52.800719 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321f1ed4-ec2e-41c8-a2f8-526cac573afa admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:52.801001 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321f1ed4-ec2e-41c8-a2f8-526cac573afa admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:52.801276 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321f1ed4-ec2e-41c8-a2f8-526cac573afa admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:52.801909 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321f1ed4-ec2e-41c8-a2f8-526cac573afa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:52.803819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321f1ed4-ec2e-41c8-a2f8-526cac573afa admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:52.813974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1173/2358] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:23:52 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:23:52.817684 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213e3ded-4a9c-402c-84ce-cb7386e0b8cc admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:52.824434 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:DELETE=10,keystone:SELECT=86,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:52.828779 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213e3ded-4a9c-402c-84ce-cb7386e0b8cc admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:52.834515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a569eb4-073d-41b4-8896-48047b4fb865 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:52.835370 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a569eb4-073d-41b4-8896-48047b4fb865 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:52.835783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a569eb4-073d-41b4-8896-48047b4fb865 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:52.836159 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a569eb4-073d-41b4-8896-48047b4fb865 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:52.836923 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a569eb4-073d-41b4-8896-48047b4fb865 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:52.838490 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a569eb4-073d-41b4-8896-48047b4fb865 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:52.849797 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1186/2359] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:23:52 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:23:52.853825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321f1ed4-ec2e-41c8-a2f8-526cac573afa admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:52.865762 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321f1ed4-ec2e-41c8-a2f8-526cac573afa admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:52.870380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639ff113-e230-4cc4-bd45-0bac5a775a53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:52.871065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639ff113-e230-4cc4-bd45-0bac5a775a53 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:52.871350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639ff113-e230-4cc4-bd45-0bac5a775a53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:52.871627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639ff113-e230-4cc4-bd45-0bac5a775a53 admin admin] PATH_INFO: `/v3/projects/0576f0261ea348ce82ebcf10fc9afab2/users/d84270ced5e74a48bcd37e924bdc98b3/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:52.885928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639ff113-e230-4cc4-bd45-0bac5a775a53 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0576f0261ea348ce82ebcf10fc9afab2, user_id=d84270ced5e74a48bcd37e924bdc98b3, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:52.889101 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639ff113-e230-4cc4-bd45-0bac5a775a53 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:52.902024 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1174/2360] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:23:52 2022] PUT /identity/v3/projects/0576f0261ea348ce82ebcf10fc9afab2/users/d84270ced5e74a48bcd37e924bdc98b3/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:52.913809 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e6a7da-af19-4fbc-b509-4f834849100b None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:52.914157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e6a7da-af19-4fbc-b509-4f834849100b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:52.914493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e6a7da-af19-4fbc-b509-4f834849100b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:52.941559 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c7e6a7da-af19-4fbc-b509-4f834849100b None None] MFA Rules not processed for user `d84270ced5e74a48bcd37e924bdc98b3`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:52.942736 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c7e6a7da-af19-4fbc-b509-4f834849100b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:53.004537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1187/2361] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:52 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:23:53.009184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639ff113-e230-4cc4-bd45-0bac5a775a53 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:53.086027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639ff113-e230-4cc4-bd45-0bac5a775a53 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:53.091313 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2d8ed4-4f69-4ec0-bd44-1dfb1639bd75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:53.091942 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2d8ed4-4f69-4ec0-bd44-1dfb1639bd75 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:53.092217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2d8ed4-4f69-4ec0-bd44-1dfb1639bd75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:53.092488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2d8ed4-4f69-4ec0-bd44-1dfb1639bd75 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:53.093104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2d8ed4-4f69-4ec0-bd44-1dfb1639bd75 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:53.095019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2d8ed4-4f69-4ec0-bd44-1dfb1639bd75 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:53.124747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1175/2362] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:23:53 2022] POST /identity/v3/projects => generated 367 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:23:53.125884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e6a7da-af19-4fbc-b509-4f834849100b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:53.137677 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e6a7da-af19-4fbc-b509-4f834849100b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:53.195700 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c136c5-be18-4037-8854-653f4e34586f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:53.196313 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c136c5-be18-4037-8854-653f4e34586f admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:53.196612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c136c5-be18-4037-8854-653f4e34586f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:53.196881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c136c5-be18-4037-8854-653f4e34586f admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:53.197502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c136c5-be18-4037-8854-653f4e34586f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:53.200085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c136c5-be18-4037-8854-653f4e34586f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:53.222188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1188/2363] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:23:53 2022] POST /identity/v3/users => generated 290 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:23:53.226096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2d8ed4-4f69-4ec0-bd44-1dfb1639bd75 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:53.236598 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2d8ed4-4f69-4ec0-bd44-1dfb1639bd75 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:53.241413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57851a8-f214-4f79-ba49-5c845209f561 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:53.242098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57851a8-f214-4f79-ba49-5c845209f561 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:53.242448 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57851a8-f214-4f79-ba49-5c845209f561 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:53.242810 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57851a8-f214-4f79-ba49-5c845209f561 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:53.243578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57851a8-f214-4f79-ba49-5c845209f561 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:53.245262 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57851a8-f214-4f79-ba49-5c845209f561 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:53.259593 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1176/2364] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:23:53 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:23:53.260779 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c136c5-be18-4037-8854-653f4e34586f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:53.270964 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57851a8-f214-4f79-ba49-5c845209f561 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:53.272613 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c136c5-be18-4037-8854-653f4e34586f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:53.272740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c57851a8-f214-4f79-ba49-5c845209f561 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:53.280319 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:53.284169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f4bd87-03c8-43be-b8b0-51376636f5f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:53.284961 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f4bd87-03c8-43be-b8b0-51376636f5f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:53.285347 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f4bd87-03c8-43be-b8b0-51376636f5f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:53.285757 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f4bd87-03c8-43be-b8b0-51376636f5f5 admin admin] PATH_INFO: `/v3/projects/d4b69e3d95c34501992dfecc4905c91e/users/5b57d899be66414a876b45be545c4e9a/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:53.301844 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f4bd87-03c8-43be-b8b0-51376636f5f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4b69e3d95c34501992dfecc4905c91e, user_id=5b57d899be66414a876b45be545c4e9a, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:53.304501 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f4bd87-03c8-43be-b8b0-51376636f5f5 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:53.319342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1189/2365] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:23:53 2022] PUT /identity/v3/projects/d4b69e3d95c34501992dfecc4905c91e/users/5b57d899be66414a876b45be545c4e9a/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:53.323062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f4bd87-03c8-43be-b8b0-51376636f5f5 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:53.355559 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57851a8-f214-4f79-ba49-5c845209f561 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:53.372713 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:53.388477 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f4bd87-03c8-43be-b8b0-51376636f5f5 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:53.393278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15612bc5-a6d0-43df-a7e0-cf6b8adae67d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:53.393886 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15612bc5-a6d0-43df-a7e0-cf6b8adae67d admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:53.394165 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15612bc5-a6d0-43df-a7e0-cf6b8adae67d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:53.394438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15612bc5-a6d0-43df-a7e0-cf6b8adae67d admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:53.395112 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15612bc5-a6d0-43df-a7e0-cf6b8adae67d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:53.396708 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15612bc5-a6d0-43df-a7e0-cf6b8adae67d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:53.411253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1190/2366] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:23:53 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:23:53.412040 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15612bc5-a6d0-43df-a7e0-cf6b8adae67d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:53.423693 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15612bc5-a6d0-43df-a7e0-cf6b8adae67d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:53.429652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ded51e-7e2b-4471-8d56-a33ef52dfc0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:53.430406 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ded51e-7e2b-4471-8d56-a33ef52dfc0a admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:53.430752 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ded51e-7e2b-4471-8d56-a33ef52dfc0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:53.431120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ded51e-7e2b-4471-8d56-a33ef52dfc0a admin admin] PATH_INFO: `/v3/projects/d4b69e3d95c34501992dfecc4905c91e/users/5b57d899be66414a876b45be545c4e9a/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:53.434011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ded51e-7e2b-4471-8d56-a33ef52dfc0a admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4b69e3d95c34501992dfecc4905c91e, user_id=5b57d899be66414a876b45be545c4e9a, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:53.437528 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ded51e-7e2b-4471-8d56-a33ef52dfc0a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:53.440513 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0b7d62-ab24-4cc2-afee-724b74040b94 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:53.441124 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0b7d62-ab24-4cc2-afee-724b74040b94 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:53.441401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0b7d62-ab24-4cc2-afee-724b74040b94 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:53.441670 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0b7d62-ab24-4cc2-afee-724b74040b94 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:53.443406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-af0b7d62-ab24-4cc2-afee-724b74040b94 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:53.448410 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1191/2367] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:23:53 2022] PUT /identity/v3/projects/d4b69e3d95c34501992dfecc4905c91e/users/5b57d899be66414a876b45be545c4e9a/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:53.459104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac06059-3172-42d4-b46b-14b209062f9c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:53.459389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac06059-3172-42d4-b46b-14b209062f9c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:53.459668 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac06059-3172-42d4-b46b-14b209062f9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:53.486892 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7ac06059-3172-42d4-b46b-14b209062f9c None None] MFA Rules not processed for user `5b57d899be66414a876b45be545c4e9a`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:53.488087 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7ac06059-3172-42d4-b46b-14b209062f9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:53.503698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0b7d62-ab24-4cc2-afee-724b74040b94 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:53.505410 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0b7d62-ab24-4cc2-afee-724b74040b94 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:53.516090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1177/2368] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:23:53 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 246 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:23:53.562631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1192/2369] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:53 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:23:53.574901 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0b7d62-ab24-4cc2-afee-724b74040b94 service nova] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:53.576327 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-af0b7d62-ab24-4cc2-afee-724b74040b94 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:53.619679 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:53.634008 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0b7d62-ab24-4cc2-afee-724b74040b94 service nova] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:53.686880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea99200-b136-4001-b6f5-52d28c3fe4b2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:53.687498 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea99200-b136-4001-b6f5-52d28c3fe4b2 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:53.687794 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea99200-b136-4001-b6f5-52d28c3fe4b2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:53.688047 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea99200-b136-4001-b6f5-52d28c3fe4b2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:53.696838 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea99200-b136-4001-b6f5-52d28c3fe4b2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:53.698250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea99200-b136-4001-b6f5-52d28c3fe4b2 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:53.707695 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1178/2370] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:23:53 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:23:55.053770 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac06059-3172-42d4-b46b-14b209062f9c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:55.116727 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac06059-3172-42d4-b46b-14b209062f9c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:55.121771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbb2b2b-a10b-4ab1-990c-7c8858a50505 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:55.122551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbb2b2b-a10b-4ab1-990c-7c8858a50505 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:55.122866 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbb2b2b-a10b-4ab1-990c-7c8858a50505 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:55.123159 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbb2b2b-a10b-4ab1-990c-7c8858a50505 admin admin] PATH_INFO: `/v3/users/d84270ced5e74a48bcd37e924bdc98b3` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:55.124281 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efbb2b2b-a10b-4ab1-990c-7c8858a50505 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d84270ced5e74a48bcd37e924bdc98b3)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:55.126349 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efbb2b2b-a10b-4ab1-990c-7c8858a50505 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:55.174111 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-efbb2b2b-a10b-4ab1-990c-7c8858a50505 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd84270ced5e74a48bcd37e924bdc98b3'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:55.174695 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-efbb2b2b-a10b-4ab1-990c-7c8858a50505 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd84270ced5e74a48bcd37e924bdc98b3'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:55.181912 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-efbb2b2b-a10b-4ab1-990c-7c8858a50505 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd84270ced5e74a48bcd37e924bdc98b3'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:55.182835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-efbb2b2b-a10b-4ab1-990c-7c8858a50505 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd84270ced5e74a48bcd37e924bdc98b3'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:55.187966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:55.196649 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-efbb2b2b-a10b-4ab1-990c-7c8858a50505 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd84270ced5e74a48bcd37e924bdc98b3'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:55.209320 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1193/2371] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:23:55 2022] DELETE /identity/v3/users/d84270ced5e74a48bcd37e924bdc98b3 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:55.213931 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9e4120-4978-4c4f-bd88-8a75b79b713b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:55.214374 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9e4120-4978-4c4f-bd88-8a75b79b713b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:55.214652 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9e4120-4978-4c4f-bd88-8a75b79b713b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:55.233872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-dc9e4120-4978-4c4f-bd88-8a75b79b713b None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:55.235115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-dc9e4120-4978-4c4f-bd88-8a75b79b713b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:55.293757 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1179/2372] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:23:55.303585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbb2b2b-a10b-4ab1-990c-7c8858a50505 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:55.305060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-efbb2b2b-a10b-4ab1-990c-7c8858a50505 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:55.364073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbb2b2b-a10b-4ab1-990c-7c8858a50505 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:55.367752 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c4ae9f-afe3-4504-838f-eb8d90c5aba4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:55.368354 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c4ae9f-afe3-4504-838f-eb8d90c5aba4 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:55.368632 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c4ae9f-afe3-4504-838f-eb8d90c5aba4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:55.368903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c4ae9f-afe3-4504-838f-eb8d90c5aba4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:55.378572 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c4ae9f-afe3-4504-838f-eb8d90c5aba4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:55.380069 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c4ae9f-afe3-4504-838f-eb8d90c5aba4 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:55.389780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1194/2373] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:23:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:23:55.664837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9e4120-4978-4c4f-bd88-8a75b79b713b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:55.666318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-dc9e4120-4978-4c4f-bd88-8a75b79b713b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:55.680489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9e4120-4978-4c4f-bd88-8a75b79b713b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:55.686325 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d099d35-0153-4782-ab7b-40f9f554e902 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:55.687179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d099d35-0153-4782-ab7b-40f9f554e902 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:55.687464 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d099d35-0153-4782-ab7b-40f9f554e902 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:55.687721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d099d35-0153-4782-ab7b-40f9f554e902 admin admin] PATH_INFO: `/v3/projects/0576f0261ea348ce82ebcf10fc9afab2` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:55.688847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d099d35-0153-4782-ab7b-40f9f554e902 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0576f0261ea348ce82ebcf10fc9afab2)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:55.690753 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d099d35-0153-4782-ab7b-40f9f554e902 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:55.737561 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9d099d35-0153-4782-ab7b-40f9f554e902 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0576f0261ea348ce82ebcf10fc9afab2', 'request_id': 'req-9d099d35-0153-4782-ab7b-40f9f554e902'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:55.742965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9d099d35-0153-4782-ab7b-40f9f554e902 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0576f0261ea348ce82ebcf10fc9afab2', 'request_id': 'req-9d099d35-0153-4782-ab7b-40f9f554e902'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:55.749608 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9d099d35-0153-4782-ab7b-40f9f554e902 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0576f0261ea348ce82ebcf10fc9afab2', 'request_id': 'req-9d099d35-0153-4782-ab7b-40f9f554e902'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:55.762333 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9d099d35-0153-4782-ab7b-40f9f554e902 admin admin] The token cache is being invalidate because project 0576f0261ea348ce82ebcf10fc9afab2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:23:55.762626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9d099d35-0153-4782-ab7b-40f9f554e902 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:55.763266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9d099d35-0153-4782-ab7b-40f9f554e902 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:55.764796 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1180/2374] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:23:55 2022] DELETE /identity/v3/projects/0576f0261ea348ce82ebcf10fc9afab2 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:55.768740 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c4ae9f-afe3-4504-838f-eb8d90c5aba4 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:55.778149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c4ae9f-afe3-4504-838f-eb8d90c5aba4 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:55.833745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83acf450-d66d-4829-96fe-027174d54418 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:55.834369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83acf450-d66d-4829-96fe-027174d54418 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:55.834647 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83acf450-d66d-4829-96fe-027174d54418 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:55.834951 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83acf450-d66d-4829-96fe-027174d54418 admin admin] PATH_INFO: `/v3/users/5b57d899be66414a876b45be545c4e9a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:55.836070 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83acf450-d66d-4829-96fe-027174d54418 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5b57d899be66414a876b45be545c4e9a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:55.837980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83acf450-d66d-4829-96fe-027174d54418 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:55.884382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-83acf450-d66d-4829-96fe-027174d54418 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5b57d899be66414a876b45be545c4e9a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:55.884519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-83acf450-d66d-4829-96fe-027174d54418 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5b57d899be66414a876b45be545c4e9a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:55.891830 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-83acf450-d66d-4829-96fe-027174d54418 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5b57d899be66414a876b45be545c4e9a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:55.892576 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-83acf450-d66d-4829-96fe-027174d54418 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5b57d899be66414a876b45be545c4e9a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:55.905551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-83acf450-d66d-4829-96fe-027174d54418 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5b57d899be66414a876b45be545c4e9a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:55.913287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:55.920390 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1195/2375] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:23:55 2022] DELETE /identity/v3/users/5b57d899be66414a876b45be545c4e9a => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:56.194007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d099d35-0153-4782-ab7b-40f9f554e902 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:56.195533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9d099d35-0153-4782-ab7b-40f9f554e902 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:56.211568 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d099d35-0153-4782-ab7b-40f9f554e902 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:56.232328 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:56.271570 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38248805-1149-4563-a822-16dc93dd6c35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:56.272134 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38248805-1149-4563-a822-16dc93dd6c35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:56.272415 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38248805-1149-4563-a822-16dc93dd6c35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:56.272689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38248805-1149-4563-a822-16dc93dd6c35 admin admin] PATH_INFO: `/v3/projects/d4b69e3d95c34501992dfecc4905c91e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:56.273811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38248805-1149-4563-a822-16dc93dd6c35 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d4b69e3d95c34501992dfecc4905c91e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:56.275755 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38248805-1149-4563-a822-16dc93dd6c35 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:56.322467 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-38248805-1149-4563-a822-16dc93dd6c35 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd4b69e3d95c34501992dfecc4905c91e', 'request_id': 'req-38248805-1149-4563-a822-16dc93dd6c35'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:56.328019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-38248805-1149-4563-a822-16dc93dd6c35 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd4b69e3d95c34501992dfecc4905c91e', 'request_id': 'req-38248805-1149-4563-a822-16dc93dd6c35'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:56.334053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-38248805-1149-4563-a822-16dc93dd6c35 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd4b69e3d95c34501992dfecc4905c91e', 'request_id': 'req-38248805-1149-4563-a822-16dc93dd6c35'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:56.346788 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-38248805-1149-4563-a822-16dc93dd6c35 admin admin] The token cache is being invalidate because project d4b69e3d95c34501992dfecc4905c91e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:23:56.347118 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-38248805-1149-4563-a822-16dc93dd6c35 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:56.347842 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-38248805-1149-4563-a822-16dc93dd6c35 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:23:56.349316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1181/2376] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:23:56 2022] DELETE /identity/v3/projects/d4b69e3d95c34501992dfecc4905c91e => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:56.354266 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092c27ef-d40a-4fdf-9f4f-ed68a07db268 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:56.354833 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092c27ef-d40a-4fdf-9f4f-ed68a07db268 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:56.354964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092c27ef-d40a-4fdf-9f4f-ed68a07db268 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:56.375060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-092c27ef-d40a-4fdf-9f4f-ed68a07db268 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:56.376320 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-092c27ef-d40a-4fdf-9f4f-ed68a07db268 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:56.436290 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1196/2377] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:23:56.444305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a36b14-34b7-4ad5-85eb-51f0c431c60b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:56.444594 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a36b14-34b7-4ad5-85eb-51f0c431c60b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:56.444875 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a36b14-34b7-4ad5-85eb-51f0c431c60b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:56.465260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-55a36b14-34b7-4ad5-85eb-51f0c431c60b None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:56.466461 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-55a36b14-34b7-4ad5-85eb-51f0c431c60b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:56.474633 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1182/2378] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:23:56.479130 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092c27ef-d40a-4fdf-9f4f-ed68a07db268 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:56.489556 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092c27ef-d40a-4fdf-9f4f-ed68a07db268 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:56.494045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d7873c-9f0e-4063-9736-2aaf4f211854 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:56.494638 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d7873c-9f0e-4063-9736-2aaf4f211854 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:56.494941 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d7873c-9f0e-4063-9736-2aaf4f211854 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:56.495212 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d7873c-9f0e-4063-9736-2aaf4f211854 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:56.495842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d7873c-9f0e-4063-9736-2aaf4f211854 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:23:56.496166 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d7873c-9f0e-4063-9736-2aaf4f211854 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:56.497684 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d7873c-9f0e-4063-9736-2aaf4f211854 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:56.514151 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1197/2379] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:23:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:23:56.515354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a36b14-34b7-4ad5-85eb-51f0c431c60b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:56.524714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a36b14-34b7-4ad5-85eb-51f0c431c60b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:56.530405 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2234ceb-9017-4de3-b244-97ade465dc1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:56.531179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2234ceb-9017-4de3-b244-97ade465dc1f admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:56.531459 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2234ceb-9017-4de3-b244-97ade465dc1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:56.531731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2234ceb-9017-4de3-b244-97ade465dc1f admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:56.532481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2234ceb-9017-4de3-b244-97ade465dc1f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:56.534369 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2234ceb-9017-4de3-b244-97ade465dc1f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:56.563559 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1183/2380] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:23:56 2022] POST /identity/v3/projects => generated 369 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:23:56.567692 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d7873c-9f0e-4063-9736-2aaf4f211854 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:56.578408 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d7873c-9f0e-4063-9736-2aaf4f211854 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:56.639441 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270523de-a65b-4aa2-89d2-a53d43c2c42d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:56.640055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270523de-a65b-4aa2-89d2-a53d43c2c42d admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:56.640330 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270523de-a65b-4aa2-89d2-a53d43c2c42d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:56.640604 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270523de-a65b-4aa2-89d2-a53d43c2c42d admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:56.641343 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270523de-a65b-4aa2-89d2-a53d43c2c42d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:56.643268 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270523de-a65b-4aa2-89d2-a53d43c2c42d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:56.670896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1198/2381] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:23:56 2022] POST /identity/v3/users => generated 285 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:23:56.675175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2234ceb-9017-4de3-b244-97ade465dc1f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:56.688944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2234ceb-9017-4de3-b244-97ade465dc1f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:56.694354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99309412-e4ed-4403-a131-11229490d150 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:56.695075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99309412-e4ed-4403-a131-11229490d150 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:56.695437 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99309412-e4ed-4403-a131-11229490d150 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:56.695781 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99309412-e4ed-4403-a131-11229490d150 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:56.696489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99309412-e4ed-4403-a131-11229490d150 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:56.698813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99309412-e4ed-4403-a131-11229490d150 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:56.727606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1184/2382] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:23:56 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:23:56.728928 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270523de-a65b-4aa2-89d2-a53d43c2c42d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:56.742421 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270523de-a65b-4aa2-89d2-a53d43c2c42d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:56.747996 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8f1557-9562-4469-be99-83620a28c494 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:56.748869 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8f1557-9562-4469-be99-83620a28c494 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:56.749304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8f1557-9562-4469-be99-83620a28c494 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:56.749736 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8f1557-9562-4469-be99-83620a28c494 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:56.751019 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8f1557-9562-4469-be99-83620a28c494 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:56.752977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8f1557-9562-4469-be99-83620a28c494 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:56.766478 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1199/2383] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:23:56 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:23:56.771086 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99309412-e4ed-4403-a131-11229490d150 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:56.786200 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99309412-e4ed-4403-a131-11229490d150 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:56.791294 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe61d53-b1f4-4096-a257-5b13e21dc6ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:56.792150 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe61d53-b1f4-4096-a257-5b13e21dc6ca admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:56.792641 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe61d53-b1f4-4096-a257-5b13e21dc6ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:56.793089 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe61d53-b1f4-4096-a257-5b13e21dc6ca admin admin] PATH_INFO: `/v3/projects/e9707bbbf71f4433bc9d0b299e90cb4f/users/b3596f963f4d436a9cfd28d40242bc33/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:56.815839 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe61d53-b1f4-4096-a257-5b13e21dc6ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9707bbbf71f4433bc9d0b299e90cb4f, user_id=b3596f963f4d436a9cfd28d40242bc33, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:56.820497 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe61d53-b1f4-4096-a257-5b13e21dc6ca admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:56.840326 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1185/2384] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:23:56 2022] PUT /identity/v3/projects/e9707bbbf71f4433bc9d0b299e90cb4f/users/b3596f963f4d436a9cfd28d40242bc33/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:56.850450 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0091d1-aa45-4be2-a14a-080366b7b7bc None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:56.850815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0091d1-aa45-4be2-a14a-080366b7b7bc None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:56.851186 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0091d1-aa45-4be2-a14a-080366b7b7bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:56.879567 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-0c0091d1-aa45-4be2-a14a-080366b7b7bc None None] MFA Rules not processed for user `b3596f963f4d436a9cfd28d40242bc33`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:56.880802 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0c0091d1-aa45-4be2-a14a-080366b7b7bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:56.937302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:56.949692 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1200/2385] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:56 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:23:56.954244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe61d53-b1f4-4096-a257-5b13e21dc6ca admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:57.003491 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:57.018375 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe61d53-b1f4-4096-a257-5b13e21dc6ca admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:57.023178 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9574fff-d277-42c9-9a16-535cb490ac4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:57.023782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9574fff-d277-42c9-9a16-535cb490ac4a admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:57.024059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9574fff-d277-42c9-9a16-535cb490ac4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:57.024326 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9574fff-d277-42c9-9a16-535cb490ac4a admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:57.024940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9574fff-d277-42c9-9a16-535cb490ac4a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:57.026831 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9574fff-d277-42c9-9a16-535cb490ac4a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:57.054904 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1186/2386] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:23:56 2022] POST /identity/v3/projects => generated 365 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:23:57.059638 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0091d1-aa45-4be2-a14a-080366b7b7bc None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:57.070517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0091d1-aa45-4be2-a14a-080366b7b7bc None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:57.131711 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2586e4-0508-4e36-8628-788cef0d1072 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:57.132385 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2586e4-0508-4e36-8628-788cef0d1072 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:57.132671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2586e4-0508-4e36-8628-788cef0d1072 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:57.132974 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2586e4-0508-4e36-8628-788cef0d1072 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:57.133623 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2586e4-0508-4e36-8628-788cef0d1072 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:57.135662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2586e4-0508-4e36-8628-788cef0d1072 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:57.161052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1201/2387] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:23:57 2022] POST /identity/v3/users => generated 289 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:23:57.165056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9574fff-d277-42c9-9a16-535cb490ac4a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:57.175663 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9574fff-d277-42c9-9a16-535cb490ac4a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:57.180434 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcd6d5f-a694-4c5b-82ae-a6a27128c14f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:57.181029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcd6d5f-a694-4c5b-82ae-a6a27128c14f admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:57.181305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcd6d5f-a694-4c5b-82ae-a6a27128c14f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:57.181573 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcd6d5f-a694-4c5b-82ae-a6a27128c14f admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:57.182468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbcd6d5f-a694-4c5b-82ae-a6a27128c14f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:57.184723 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbcd6d5f-a694-4c5b-82ae-a6a27128c14f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:57.196234 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1187/2388] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:23:57 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:23:57.200969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2586e4-0508-4e36-8628-788cef0d1072 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:57.211421 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2586e4-0508-4e36-8628-788cef0d1072 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:57.216094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9979dea4-7010-4a92-b712-de3732ae7352 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:57.216772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9979dea4-7010-4a92-b712-de3732ae7352 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:57.217087 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9979dea4-7010-4a92-b712-de3732ae7352 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:57.217381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9979dea4-7010-4a92-b712-de3732ae7352 admin admin] PATH_INFO: `/v3/projects/9238c0e7855745c7a9e5919532a474fc/users/47cb538f279f4872b890dbff860a5a3b/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:57.237676 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9979dea4-7010-4a92-b712-de3732ae7352 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9238c0e7855745c7a9e5919532a474fc, user_id=47cb538f279f4872b890dbff860a5a3b, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:57.240373 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9979dea4-7010-4a92-b712-de3732ae7352 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:57.253990 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1202/2389] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:23:57 2022] PUT /identity/v3/projects/9238c0e7855745c7a9e5919532a474fc/users/47cb538f279f4872b890dbff860a5a3b/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:23:57.257984 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcd6d5f-a694-4c5b-82ae-a6a27128c14f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:57.268546 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcd6d5f-a694-4c5b-82ae-a6a27128c14f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:57.324418 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363474eb-8ea3-42b9-8c25-03475c3eac3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:57.324968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363474eb-8ea3-42b9-8c25-03475c3eac3d admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:57.325248 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363474eb-8ea3-42b9-8c25-03475c3eac3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:57.325519 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363474eb-8ea3-42b9-8c25-03475c3eac3d admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:57.326152 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363474eb-8ea3-42b9-8c25-03475c3eac3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:57.327776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363474eb-8ea3-42b9-8c25-03475c3eac3d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:57.337350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1188/2390] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:23:57 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:23:57.341451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9979dea4-7010-4a92-b712-de3732ae7352 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:57.351573 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9979dea4-7010-4a92-b712-de3732ae7352 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:57.356675 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197a1ac5-b5ea-4140-b3dd-94d6858bc5f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:57.357313 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197a1ac5-b5ea-4140-b3dd-94d6858bc5f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:57.357580 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197a1ac5-b5ea-4140-b3dd-94d6858bc5f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:57.357849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197a1ac5-b5ea-4140-b3dd-94d6858bc5f2 admin admin] PATH_INFO: `/v3/projects/9238c0e7855745c7a9e5919532a474fc/users/47cb538f279f4872b890dbff860a5a3b/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:57.359754 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197a1ac5-b5ea-4140-b3dd-94d6858bc5f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9238c0e7855745c7a9e5919532a474fc, user_id=47cb538f279f4872b890dbff860a5a3b, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:57.362369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197a1ac5-b5ea-4140-b3dd-94d6858bc5f2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:57.372997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1203/2391] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:23:57 2022] PUT /identity/v3/projects/9238c0e7855745c7a9e5919532a474fc/users/47cb538f279f4872b890dbff860a5a3b/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:23:57.382878 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fb5da3-b4b1-4885-abef-b8834fa0c7d7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:57.383176 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fb5da3-b4b1-4885-abef-b8834fa0c7d7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:57.383458 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fb5da3-b4b1-4885-abef-b8834fa0c7d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:57.410868 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-e3fb5da3-b4b1-4885-abef-b8834fa0c7d7 None None] MFA Rules not processed for user `47cb538f279f4872b890dbff860a5a3b`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:23:57.412101 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e3fb5da3-b4b1-4885-abef-b8834fa0c7d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:57.479493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1189/2392] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:23:57 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:23:57.491466 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197a1ac5-b5ea-4140-b3dd-94d6858bc5f2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:57.493198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-197a1ac5-b5ea-4140-b3dd-94d6858bc5f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:57.550787 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197a1ac5-b5ea-4140-b3dd-94d6858bc5f2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:57.554404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068ca952-f900-4903-8157-c23dbe24609f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:57.555027 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068ca952-f900-4903-8157-c23dbe24609f service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:57.555307 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068ca952-f900-4903-8157-c23dbe24609f service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:57.555579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068ca952-f900-4903-8157-c23dbe24609f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:57.565404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068ca952-f900-4903-8157-c23dbe24609f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:57.566808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068ca952-f900-4903-8157-c23dbe24609f service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:57.576185 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1204/2393] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:23:57 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:23:57.620452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fb5da3-b4b1-4885-abef-b8834fa0c7d7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:23:57.622436 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e3fb5da3-b4b1-4885-abef-b8834fa0c7d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:23:57.634344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fb5da3-b4b1-4885-abef-b8834fa0c7d7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:23:57.637899 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8470693a-0f52-49dc-a85d-d0c8bd5f9403 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:23:57.638520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8470693a-0f52-49dc-a85d-d0c8bd5f9403 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:23:57.638796 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8470693a-0f52-49dc-a85d-d0c8bd5f9403 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:23:57.639097 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8470693a-0f52-49dc-a85d-d0c8bd5f9403 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:23:57.680569 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:23:57.697203 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8470693a-0f52-49dc-a85d-d0c8bd5f9403 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:23:57.698619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8470693a-0f52-49dc-a85d-d0c8bd5f9403 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:23:57.707297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1190/2394] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:23:57 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:24:01.091625 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068ca952-f900-4903-8157-c23dbe24609f service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:01.139463 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:01.153740 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068ca952-f900-4903-8157-c23dbe24609f service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:01.158514 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33facd9-2641-4a55-9104-2495d709d05a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:01.159159 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33facd9-2641-4a55-9104-2495d709d05a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:01.159440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33facd9-2641-4a55-9104-2495d709d05a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:01.159706 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33facd9-2641-4a55-9104-2495d709d05a admin admin] PATH_INFO: `/v3/users/b3596f963f4d436a9cfd28d40242bc33` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:01.160799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33facd9-2641-4a55-9104-2495d709d05a admin admin] RBAC: Authorizing `identity:delete_user(user_id=b3596f963f4d436a9cfd28d40242bc33)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:01.162705 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33facd9-2641-4a55-9104-2495d709d05a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:01.207048 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e33facd9-2641-4a55-9104-2495d709d05a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b3596f963f4d436a9cfd28d40242bc33'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:01.207687 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e33facd9-2641-4a55-9104-2495d709d05a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b3596f963f4d436a9cfd28d40242bc33'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:01.214570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e33facd9-2641-4a55-9104-2495d709d05a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b3596f963f4d436a9cfd28d40242bc33'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:01.215242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e33facd9-2641-4a55-9104-2495d709d05a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b3596f963f4d436a9cfd28d40242bc33'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:01.227122 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e33facd9-2641-4a55-9104-2495d709d05a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b3596f963f4d436a9cfd28d40242bc33'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:01.239494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1205/2395] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:24:01 2022] DELETE /identity/v3/users/b3596f963f4d436a9cfd28d40242bc33 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:24:01.243925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83376fa0-5896-4681-a47b-e43626149a2e None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:01.244417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83376fa0-5896-4681-a47b-e43626149a2e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:01.244616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83376fa0-5896-4681-a47b-e43626149a2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:01.263954 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-83376fa0-5896-4681-a47b-e43626149a2e None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:01.265132 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-83376fa0-5896-4681-a47b-e43626149a2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:01.322428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1191/2396] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:24:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:24:01.332325 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33facd9-2641-4a55-9104-2495d709d05a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:01.333754 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e33facd9-2641-4a55-9104-2495d709d05a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:01.390633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33facd9-2641-4a55-9104-2495d709d05a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:01.400043 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b815f850-d0a7-4c40-ac0a-47f7df876536 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:01.400672 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b815f850-d0a7-4c40-ac0a-47f7df876536 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:01.400951 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b815f850-d0a7-4c40-ac0a-47f7df876536 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:01.401222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b815f850-d0a7-4c40-ac0a-47f7df876536 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:01.411690 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b815f850-d0a7-4c40-ac0a-47f7df876536 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:01.413117 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b815f850-d0a7-4c40-ac0a-47f7df876536 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:01.423451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1206/2397] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:24:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:24:01.710729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83376fa0-5896-4681-a47b-e43626149a2e None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:01.712340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-83376fa0-5896-4681-a47b-e43626149a2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:01.729733 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83376fa0-5896-4681-a47b-e43626149a2e None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:01.734490 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d8cede-9873-485f-ad5d-1582e38dd20f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:01.735179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d8cede-9873-485f-ad5d-1582e38dd20f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:01.735462 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d8cede-9873-485f-ad5d-1582e38dd20f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:01.735737 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d8cede-9873-485f-ad5d-1582e38dd20f admin admin] PATH_INFO: `/v3/projects/e9707bbbf71f4433bc9d0b299e90cb4f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:01.737047 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d8cede-9873-485f-ad5d-1582e38dd20f admin admin] RBAC: Authorizing `identity:delete_project(project_id=e9707bbbf71f4433bc9d0b299e90cb4f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:01.739265 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d8cede-9873-485f-ad5d-1582e38dd20f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:01.786594 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-69d8cede-9873-485f-ad5d-1582e38dd20f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e9707bbbf71f4433bc9d0b299e90cb4f', 'request_id': 'req-69d8cede-9873-485f-ad5d-1582e38dd20f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:01.792130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-69d8cede-9873-485f-ad5d-1582e38dd20f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e9707bbbf71f4433bc9d0b299e90cb4f', 'request_id': 'req-69d8cede-9873-485f-ad5d-1582e38dd20f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:01.798512 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-69d8cede-9873-485f-ad5d-1582e38dd20f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e9707bbbf71f4433bc9d0b299e90cb4f', 'request_id': 'req-69d8cede-9873-485f-ad5d-1582e38dd20f'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:01.813801 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-69d8cede-9873-485f-ad5d-1582e38dd20f admin admin] The token cache is being invalidate because project e9707bbbf71f4433bc9d0b299e90cb4f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:24:01.814231 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-69d8cede-9873-485f-ad5d-1582e38dd20f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:01.815005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-69d8cede-9873-485f-ad5d-1582e38dd20f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:01.816618 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1192/2398] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:24:01 2022] DELETE /identity/v3/projects/e9707bbbf71f4433bc9d0b299e90cb4f => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:24:01.820494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b815f850-d0a7-4c40-ac0a-47f7df876536 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:01.835263 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b815f850-d0a7-4c40-ac0a-47f7df876536 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:01.914214 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b8ba93-3d95-4afd-8e14-0038c0266fb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:01.915065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b8ba93-3d95-4afd-8e14-0038c0266fb0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:01.915448 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b8ba93-3d95-4afd-8e14-0038c0266fb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:01.915888 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b8ba93-3d95-4afd-8e14-0038c0266fb0 admin admin] PATH_INFO: `/v3/users/47cb538f279f4872b890dbff860a5a3b` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:01.918004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b8ba93-3d95-4afd-8e14-0038c0266fb0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=47cb538f279f4872b890dbff860a5a3b)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:01.920301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b8ba93-3d95-4afd-8e14-0038c0266fb0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:01.934959 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:01.984531 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-39b8ba93-3d95-4afd-8e14-0038c0266fb0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '47cb538f279f4872b890dbff860a5a3b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:01.985380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-39b8ba93-3d95-4afd-8e14-0038c0266fb0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '47cb538f279f4872b890dbff860a5a3b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:01.994696 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-39b8ba93-3d95-4afd-8e14-0038c0266fb0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '47cb538f279f4872b890dbff860a5a3b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:01.996322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-39b8ba93-3d95-4afd-8e14-0038c0266fb0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '47cb538f279f4872b890dbff860a5a3b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:02.009599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-39b8ba93-3d95-4afd-8e14-0038c0266fb0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '47cb538f279f4872b890dbff860a5a3b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:02.022301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1207/2399] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:24:01 2022] DELETE /identity/v3/users/47cb538f279f4872b890dbff860a5a3b => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:24:02.300146 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d8cede-9873-485f-ad5d-1582e38dd20f admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:02.301860 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-69d8cede-9873-485f-ad5d-1582e38dd20f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:02.317966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d8cede-9873-485f-ad5d-1582e38dd20f admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:02.376081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c712a7dd-1413-43f8-90b6-875f663e4d0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:02.376638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c712a7dd-1413-43f8-90b6-875f663e4d0d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:02.376916 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c712a7dd-1413-43f8-90b6-875f663e4d0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:02.377191 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c712a7dd-1413-43f8-90b6-875f663e4d0d admin admin] PATH_INFO: `/v3/projects/9238c0e7855745c7a9e5919532a474fc` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:02.378310 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c712a7dd-1413-43f8-90b6-875f663e4d0d admin admin] RBAC: Authorizing `identity:delete_project(project_id=9238c0e7855745c7a9e5919532a474fc)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:02.380292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c712a7dd-1413-43f8-90b6-875f663e4d0d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:02.388070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:02.428658 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c712a7dd-1413-43f8-90b6-875f663e4d0d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9238c0e7855745c7a9e5919532a474fc', 'request_id': 'req-c712a7dd-1413-43f8-90b6-875f663e4d0d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:02.434415 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c712a7dd-1413-43f8-90b6-875f663e4d0d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9238c0e7855745c7a9e5919532a474fc', 'request_id': 'req-c712a7dd-1413-43f8-90b6-875f663e4d0d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:02.440863 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c712a7dd-1413-43f8-90b6-875f663e4d0d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9238c0e7855745c7a9e5919532a474fc', 'request_id': 'req-c712a7dd-1413-43f8-90b6-875f663e4d0d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:02.454413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c712a7dd-1413-43f8-90b6-875f663e4d0d admin admin] The token cache is being invalidate because project 9238c0e7855745c7a9e5919532a474fc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:24:02.454698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c712a7dd-1413-43f8-90b6-875f663e4d0d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:02.455367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c712a7dd-1413-43f8-90b6-875f663e4d0d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:02.456844 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1193/2400] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:24:02 2022] DELETE /identity/v3/projects/9238c0e7855745c7a9e5919532a474fc => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:24:02.461627 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fe62a5-d755-4dc2-ad5e-c39380f5d0e2 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:02.462047 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fe62a5-d755-4dc2-ad5e-c39380f5d0e2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:02.462371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fe62a5-d755-4dc2-ad5e-c39380f5d0e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:02.483225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-39fe62a5-d755-4dc2-ad5e-c39380f5d0e2 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:02.484421 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-39fe62a5-d755-4dc2-ad5e-c39380f5d0e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:02.543433 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1208/2401] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:24:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:24:02.552131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898e6e3e-3f3a-4ad9-9a11-77bd4bf5c5e7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:02.552416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898e6e3e-3f3a-4ad9-9a11-77bd4bf5c5e7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:02.552710 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898e6e3e-3f3a-4ad9-9a11-77bd4bf5c5e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:02.572756 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-898e6e3e-3f3a-4ad9-9a11-77bd4bf5c5e7 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:02.574499 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-898e6e3e-3f3a-4ad9-9a11-77bd4bf5c5e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:02.583225 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1194/2402] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:24:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:24:02.587379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fe62a5-d755-4dc2-ad5e-c39380f5d0e2 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:02.597539 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fe62a5-d755-4dc2-ad5e-c39380f5d0e2 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:02.602067 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43d1d07-5087-455a-915e-a5257d5f1e8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:02.602668 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43d1d07-5087-455a-915e-a5257d5f1e8d admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:02.602956 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43d1d07-5087-455a-915e-a5257d5f1e8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:02.603226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43d1d07-5087-455a-915e-a5257d5f1e8d admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:02.603861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43d1d07-5087-455a-915e-a5257d5f1e8d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:24:02.604162 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43d1d07-5087-455a-915e-a5257d5f1e8d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:02.605753 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43d1d07-5087-455a-915e-a5257d5f1e8d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:02.617049 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1209/2403] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:24:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:24:02.621246 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898e6e3e-3f3a-4ad9-9a11-77bd4bf5c5e7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:02.630604 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898e6e3e-3f3a-4ad9-9a11-77bd4bf5c5e7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:02.635158 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1447b8d4-a599-4ad1-919e-1118ded126e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:02.635752 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1447b8d4-a599-4ad1-919e-1118ded126e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:02.636529 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1447b8d4-a599-4ad1-919e-1118ded126e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:02.636802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1447b8d4-a599-4ad1-919e-1118ded126e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:02.637430 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1447b8d4-a599-4ad1-919e-1118ded126e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:02.639361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1447b8d4-a599-4ad1-919e-1118ded126e9 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:02.671493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1195/2404] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:24:02 2022] POST /identity/v3/projects => generated 385 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:24:02.672701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43d1d07-5087-455a-915e-a5257d5f1e8d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:02.682951 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43d1d07-5087-455a-915e-a5257d5f1e8d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:02.740849 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcaa7cbf-69ed-4fa4-ac05-0c91cdefbb07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:02.741446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcaa7cbf-69ed-4fa4-ac05-0c91cdefbb07 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:02.741724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcaa7cbf-69ed-4fa4-ac05-0c91cdefbb07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:02.741991 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcaa7cbf-69ed-4fa4-ac05-0c91cdefbb07 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:02.742613 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcaa7cbf-69ed-4fa4-ac05-0c91cdefbb07 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:02.744557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcaa7cbf-69ed-4fa4-ac05-0c91cdefbb07 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:02.765911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1210/2405] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:24:02 2022] POST /identity/v3/users => generated 293 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:24:02.769981 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1447b8d4-a599-4ad1-919e-1118ded126e9 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:02.780747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1447b8d4-a599-4ad1-919e-1118ded126e9 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:02.785327 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc573837-c979-42be-9fa2-8685ccb2a9f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:02.785926 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc573837-c979-42be-9fa2-8685ccb2a9f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:02.786197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc573837-c979-42be-9fa2-8685ccb2a9f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:02.786462 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc573837-c979-42be-9fa2-8685ccb2a9f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:02.787119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc573837-c979-42be-9fa2-8685ccb2a9f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:02.788691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc573837-c979-42be-9fa2-8685ccb2a9f2 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:02.798842 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1196/2406] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:24:02 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:24:02.802963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcaa7cbf-69ed-4fa4-ac05-0c91cdefbb07 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:02.813351 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcaa7cbf-69ed-4fa4-ac05-0c91cdefbb07 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:02.818048 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2890aa2c-54b8-4f6a-ad15-9de6e9fb4888 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:02.818719 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2890aa2c-54b8-4f6a-ad15-9de6e9fb4888 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:02.819039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2890aa2c-54b8-4f6a-ad15-9de6e9fb4888 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:02.819401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2890aa2c-54b8-4f6a-ad15-9de6e9fb4888 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:02.820227 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2890aa2c-54b8-4f6a-ad15-9de6e9fb4888 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:02.821874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2890aa2c-54b8-4f6a-ad15-9de6e9fb4888 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:02.823197 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:02.836420 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1211/2407] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:24:02 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:24:02.841026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc573837-c979-42be-9fa2-8685ccb2a9f2 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:02.854568 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc573837-c979-42be-9fa2-8685ccb2a9f2 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:02.861349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5457ab4-5cb8-4fdd-ab1e-0d9927a35459 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:02.862085 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5457ab4-5cb8-4fdd-ab1e-0d9927a35459 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:02.862427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5457ab4-5cb8-4fdd-ab1e-0d9927a35459 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:02.862706 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5457ab4-5cb8-4fdd-ab1e-0d9927a35459 admin admin] PATH_INFO: `/v3/projects/338f0b464255481b889f50459f925f23/users/fcba4763051b41309dac06071c3da6e7/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:02.877931 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5457ab4-5cb8-4fdd-ab1e-0d9927a35459 admin admin] RBAC: Authorizing `identity:create_grant(project_id=338f0b464255481b889f50459f925f23, user_id=fcba4763051b41309dac06071c3da6e7, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:02.880591 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5457ab4-5cb8-4fdd-ab1e-0d9927a35459 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:02.895938 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1197/2408] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:24:02 2022] PUT /identity/v3/projects/338f0b464255481b889f50459f925f23/users/fcba4763051b41309dac06071c3da6e7/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:24:02.906912 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08241f4-626e-4aaf-b746-e0e85abcf375 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:02.907201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08241f4-626e-4aaf-b746-e0e85abcf375 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:02.907476 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08241f4-626e-4aaf-b746-e0e85abcf375 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:02.936966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a08241f4-626e-4aaf-b746-e0e85abcf375 None None] MFA Rules not processed for user `fcba4763051b41309dac06071c3da6e7`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:02.938159 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a08241f4-626e-4aaf-b746-e0e85abcf375 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:03.006712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1212/2409] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:24:02 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:24:03.011821 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5457ab4-5cb8-4fdd-ab1e-0d9927a35459 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:03.083918 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5457ab4-5cb8-4fdd-ab1e-0d9927a35459 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:03.091508 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95db88a0-4e35-46b4-ad5c-34fb75818346 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:03.092125 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95db88a0-4e35-46b4-ad5c-34fb75818346 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:03.092425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95db88a0-4e35-46b4-ad5c-34fb75818346 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:03.093557 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95db88a0-4e35-46b4-ad5c-34fb75818346 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:03.094214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95db88a0-4e35-46b4-ad5c-34fb75818346 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:03.096302 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95db88a0-4e35-46b4-ad5c-34fb75818346 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:03.126906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1198/2410] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:24:03 2022] POST /identity/v3/projects => generated 385 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:24:03.131495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08241f4-626e-4aaf-b746-e0e85abcf375 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:03.144277 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08241f4-626e-4aaf-b746-e0e85abcf375 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:03.232159 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1ee553-242b-47f8-88ac-3ece4fbb22cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:03.232789 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1ee553-242b-47f8-88ac-3ece4fbb22cc admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:03.233069 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1ee553-242b-47f8-88ac-3ece4fbb22cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:03.233341 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1ee553-242b-47f8-88ac-3ece4fbb22cc admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:03.233994 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1ee553-242b-47f8-88ac-3ece4fbb22cc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:03.235970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1ee553-242b-47f8-88ac-3ece4fbb22cc admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:03.259469 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1213/2411] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:24:03 2022] POST /identity/v3/users => generated 299 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:24:03.264257 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95db88a0-4e35-46b4-ad5c-34fb75818346 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:03.275530 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95db88a0-4e35-46b4-ad5c-34fb75818346 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:03.280919 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c700c7-82d4-4d9b-9d50-65a607dba04b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:03.281636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c700c7-82d4-4d9b-9d50-65a607dba04b admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:03.281984 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c700c7-82d4-4d9b-9d50-65a607dba04b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:03.282328 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c700c7-82d4-4d9b-9d50-65a607dba04b admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:03.283080 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c700c7-82d4-4d9b-9d50-65a607dba04b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:03.284770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c700c7-82d4-4d9b-9d50-65a607dba04b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:03.297011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1199/2412] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:24:03 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:24:03.300927 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1ee553-242b-47f8-88ac-3ece4fbb22cc admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:03.312231 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1ee553-242b-47f8-88ac-3ece4fbb22cc admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:03.318447 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af90782-0167-4d41-879d-4cd9380a6a6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:03.319495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af90782-0167-4d41-879d-4cd9380a6a6f admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:03.319882 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af90782-0167-4d41-879d-4cd9380a6a6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:03.320257 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af90782-0167-4d41-879d-4cd9380a6a6f admin admin] PATH_INFO: `/v3/projects/afc20633acd749a1b3aa1d36dd3d40ef/users/757df6ddd00a4565bcb795e4539432ee/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:03.337249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af90782-0167-4d41-879d-4cd9380a6a6f admin admin] RBAC: Authorizing `identity:create_grant(project_id=afc20633acd749a1b3aa1d36dd3d40ef, user_id=757df6ddd00a4565bcb795e4539432ee, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:03.340149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af90782-0167-4d41-879d-4cd9380a6a6f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:03.356554 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1214/2413] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:24:03 2022] PUT /identity/v3/projects/afc20633acd749a1b3aa1d36dd3d40ef/users/757df6ddd00a4565bcb795e4539432ee/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:24:03.360511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c700c7-82d4-4d9b-9d50-65a607dba04b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:03.372297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c700c7-82d4-4d9b-9d50-65a607dba04b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:03.377085 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:03.439943 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d258ca5-e917-47eb-a564-559d00e106c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:03.440601 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d258ca5-e917-47eb-a564-559d00e106c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:03.440904 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d258ca5-e917-47eb-a564-559d00e106c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:03.441202 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d258ca5-e917-47eb-a564-559d00e106c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:03.441847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d258ca5-e917-47eb-a564-559d00e106c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:03.443515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d258ca5-e917-47eb-a564-559d00e106c4 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:03.454054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1200/2414] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:24:03 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:24:03.457821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af90782-0167-4d41-879d-4cd9380a6a6f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:03.468234 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af90782-0167-4d41-879d-4cd9380a6a6f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:03.472924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f2f7b4-0b43-4f95-9099-578d5713831b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:03.473709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f2f7b4-0b43-4f95-9099-578d5713831b admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:03.473992 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f2f7b4-0b43-4f95-9099-578d5713831b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:03.474257 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f2f7b4-0b43-4f95-9099-578d5713831b admin admin] PATH_INFO: `/v3/projects/afc20633acd749a1b3aa1d36dd3d40ef/users/757df6ddd00a4565bcb795e4539432ee/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:03.476183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f2f7b4-0b43-4f95-9099-578d5713831b admin admin] RBAC: Authorizing `identity:create_grant(project_id=afc20633acd749a1b3aa1d36dd3d40ef, user_id=757df6ddd00a4565bcb795e4539432ee, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:03.478764 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f2f7b4-0b43-4f95-9099-578d5713831b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:03.492793 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1215/2415] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:24:03 2022] PUT /identity/v3/projects/afc20633acd749a1b3aa1d36dd3d40ef/users/757df6ddd00a4565bcb795e4539432ee/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:24:03.512221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3478f9-7ee4-4ad2-b539-bb9c52b26b8a None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:03.512517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3478f9-7ee4-4ad2-b539-bb9c52b26b8a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:03.512834 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3478f9-7ee4-4ad2-b539-bb9c52b26b8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:03.543640 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9d3478f9-7ee4-4ad2-b539-bb9c52b26b8a None None] MFA Rules not processed for user `757df6ddd00a4565bcb795e4539432ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:03.544538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9d3478f9-7ee4-4ad2-b539-bb9c52b26b8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:03.624500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1201/2416] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:24:03 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:24:03.636178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f2f7b4-0b43-4f95-9099-578d5713831b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:03.637977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a4f2f7b4-0b43-4f95-9099-578d5713831b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:03.681335 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:03.698704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f2f7b4-0b43-4f95-9099-578d5713831b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:03.703010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8d9ace-ac04-4660-bbfd-5c9a143dcdc5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:03.703633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8d9ace-ac04-4660-bbfd-5c9a143dcdc5 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:03.704032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8d9ace-ac04-4660-bbfd-5c9a143dcdc5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:03.704307 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8d9ace-ac04-4660-bbfd-5c9a143dcdc5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:03.713883 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8d9ace-ac04-4660-bbfd-5c9a143dcdc5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:03.716031 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8d9ace-ac04-4660-bbfd-5c9a143dcdc5 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:03.725843 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1216/2417] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:24:03 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:24:03.795388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3478f9-7ee4-4ad2-b539-bb9c52b26b8a None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:03.858939 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3478f9-7ee4-4ad2-b539-bb9c52b26b8a None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:03.863509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d25914-f3e6-4d9c-a9fa-c3b952a3a71e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:03.864152 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d25914-f3e6-4d9c-a9fa-c3b952a3a71e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:03.864455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d25914-f3e6-4d9c-a9fa-c3b952a3a71e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:03.864726 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d25914-f3e6-4d9c-a9fa-c3b952a3a71e admin admin] PATH_INFO: `/v3/users/fcba4763051b41309dac06071c3da6e7` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:03.865848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d25914-f3e6-4d9c-a9fa-c3b952a3a71e admin admin] RBAC: Authorizing `identity:delete_user(user_id=fcba4763051b41309dac06071c3da6e7)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:03.867807 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d25914-f3e6-4d9c-a9fa-c3b952a3a71e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:03.903163 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:03.918054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e5d25914-f3e6-4d9c-a9fa-c3b952a3a71e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fcba4763051b41309dac06071c3da6e7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:03.918643 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e5d25914-f3e6-4d9c-a9fa-c3b952a3a71e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fcba4763051b41309dac06071c3da6e7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:03.926248 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e5d25914-f3e6-4d9c-a9fa-c3b952a3a71e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fcba4763051b41309dac06071c3da6e7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:03.926876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e5d25914-f3e6-4d9c-a9fa-c3b952a3a71e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fcba4763051b41309dac06071c3da6e7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:03.935474 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e5d25914-f3e6-4d9c-a9fa-c3b952a3a71e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fcba4763051b41309dac06071c3da6e7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:03.953342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1202/2418] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:24:03 2022] DELETE /identity/v3/users/fcba4763051b41309dac06071c3da6e7 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:24:03.957713 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62bc002-0f8e-42b6-a05c-ee6dfa1273ab None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:03.958001 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62bc002-0f8e-42b6-a05c-ee6dfa1273ab None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:03.958299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62bc002-0f8e-42b6-a05c-ee6dfa1273ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:03.977248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e62bc002-0f8e-42b6-a05c-ee6dfa1273ab None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:03.978436 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e62bc002-0f8e-42b6-a05c-ee6dfa1273ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:04.037234 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1217/2419] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:24:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:24:04.049068 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d25914-f3e6-4d9c-a9fa-c3b952a3a71e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:04.050512 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e5d25914-f3e6-4d9c-a9fa-c3b952a3a71e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:04.109921 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d25914-f3e6-4d9c-a9fa-c3b952a3a71e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:04.114187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b98aef-5dc4-4461-bda5-44441e2213eb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:04.114872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b98aef-5dc4-4461-bda5-44441e2213eb service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:04.115188 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b98aef-5dc4-4461-bda5-44441e2213eb service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:04.115437 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b98aef-5dc4-4461-bda5-44441e2213eb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:04.125378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b98aef-5dc4-4461-bda5-44441e2213eb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:04.126923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b98aef-5dc4-4461-bda5-44441e2213eb service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:04.137619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1203/2420] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:24:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:24:04.690665 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62bc002-0f8e-42b6-a05c-ee6dfa1273ab None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:04.692130 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e62bc002-0f8e-42b6-a05c-ee6dfa1273ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:04.723275 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62bc002-0f8e-42b6-a05c-ee6dfa1273ab None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:04.730040 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92503dd3-3149-4b8b-9b4f-9d9765236a10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:04.731051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92503dd3-3149-4b8b-9b4f-9d9765236a10 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:04.731427 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92503dd3-3149-4b8b-9b4f-9d9765236a10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:04.732139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92503dd3-3149-4b8b-9b4f-9d9765236a10 admin admin] PATH_INFO: `/v3/projects/338f0b464255481b889f50459f925f23` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:04.733371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92503dd3-3149-4b8b-9b4f-9d9765236a10 admin admin] RBAC: Authorizing `identity:delete_project(project_id=338f0b464255481b889f50459f925f23)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:04.735509 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92503dd3-3149-4b8b-9b4f-9d9765236a10 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:04.792682 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-92503dd3-3149-4b8b-9b4f-9d9765236a10 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '338f0b464255481b889f50459f925f23', 'request_id': 'req-92503dd3-3149-4b8b-9b4f-9d9765236a10'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:04.805806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-92503dd3-3149-4b8b-9b4f-9d9765236a10 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '338f0b464255481b889f50459f925f23', 'request_id': 'req-92503dd3-3149-4b8b-9b4f-9d9765236a10'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:04.816141 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-92503dd3-3149-4b8b-9b4f-9d9765236a10 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '338f0b464255481b889f50459f925f23', 'request_id': 'req-92503dd3-3149-4b8b-9b4f-9d9765236a10'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:04.835139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-92503dd3-3149-4b8b-9b4f-9d9765236a10 admin admin] The token cache is being invalidate because project 338f0b464255481b889f50459f925f23 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:24:04.835431 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-92503dd3-3149-4b8b-9b4f-9d9765236a10 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:04.836100 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-92503dd3-3149-4b8b-9b4f-9d9765236a10 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:04.837563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1218/2421] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:24:04 2022] DELETE /identity/v3/projects/338f0b464255481b889f50459f925f23 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:24:04.844417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b98aef-5dc4-4461-bda5-44441e2213eb service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:04.858715 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b98aef-5dc4-4461-bda5-44441e2213eb service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:04.922108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea07c75a-8cd1-4f0e-855a-0d3bc083a9d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:04.922760 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea07c75a-8cd1-4f0e-855a-0d3bc083a9d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:04.923072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea07c75a-8cd1-4f0e-855a-0d3bc083a9d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:04.923343 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea07c75a-8cd1-4f0e-855a-0d3bc083a9d8 admin admin] PATH_INFO: `/v3/users/757df6ddd00a4565bcb795e4539432ee` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:04.924483 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea07c75a-8cd1-4f0e-855a-0d3bc083a9d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=757df6ddd00a4565bcb795e4539432ee)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:04.926416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea07c75a-8cd1-4f0e-855a-0d3bc083a9d8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:04.976614 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ea07c75a-8cd1-4f0e-855a-0d3bc083a9d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '757df6ddd00a4565bcb795e4539432ee'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:04.977388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ea07c75a-8cd1-4f0e-855a-0d3bc083a9d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '757df6ddd00a4565bcb795e4539432ee'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:04.981530 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:04.985969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ea07c75a-8cd1-4f0e-855a-0d3bc083a9d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '757df6ddd00a4565bcb795e4539432ee'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:04.987132 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ea07c75a-8cd1-4f0e-855a-0d3bc083a9d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '757df6ddd00a4565bcb795e4539432ee'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:04.995212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ea07c75a-8cd1-4f0e-855a-0d3bc083a9d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '757df6ddd00a4565bcb795e4539432ee'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:05.016278 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1204/2422] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:24:04 2022] DELETE /identity/v3/users/757df6ddd00a4565bcb795e4539432ee => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:24:05.427658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92503dd3-3149-4b8b-9b4f-9d9765236a10 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:05.429308 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-92503dd3-3149-4b8b-9b4f-9d9765236a10 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:05.443565 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92503dd3-3149-4b8b-9b4f-9d9765236a10 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:05.531681 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f702f836-e255-4903-84f0-0c43bdd44d7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:05.533715 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f702f836-e255-4903-84f0-0c43bdd44d7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:05.536935 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f702f836-e255-4903-84f0-0c43bdd44d7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:05.537498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f702f836-e255-4903-84f0-0c43bdd44d7f admin admin] PATH_INFO: `/v3/projects/afc20633acd749a1b3aa1d36dd3d40ef` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:05.539395 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f702f836-e255-4903-84f0-0c43bdd44d7f admin admin] RBAC: Authorizing `identity:delete_project(project_id=afc20633acd749a1b3aa1d36dd3d40ef)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:05.542282 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f702f836-e255-4903-84f0-0c43bdd44d7f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:05.553750 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:05.611823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f702f836-e255-4903-84f0-0c43bdd44d7f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'afc20633acd749a1b3aa1d36dd3d40ef', 'request_id': 'req-f702f836-e255-4903-84f0-0c43bdd44d7f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:05.621332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f702f836-e255-4903-84f0-0c43bdd44d7f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'afc20633acd749a1b3aa1d36dd3d40ef', 'request_id': 'req-f702f836-e255-4903-84f0-0c43bdd44d7f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:05.629246 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f702f836-e255-4903-84f0-0c43bdd44d7f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'afc20633acd749a1b3aa1d36dd3d40ef', 'request_id': 'req-f702f836-e255-4903-84f0-0c43bdd44d7f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:05.654136 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f702f836-e255-4903-84f0-0c43bdd44d7f admin admin] The token cache is being invalidate because project afc20633acd749a1b3aa1d36dd3d40ef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:24:05.654643 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f702f836-e255-4903-84f0-0c43bdd44d7f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:05.655572 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f702f836-e255-4903-84f0-0c43bdd44d7f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:05.673821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1219/2423] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:24:05 2022] DELETE /identity/v3/projects/afc20633acd749a1b3aa1d36dd3d40ef => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:24:05.676445 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3d319c-d7cd-4697-8b29-4ff3d51718ba None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:05.676730 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3d319c-d7cd-4697-8b29-4ff3d51718ba None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:05.676999 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3d319c-d7cd-4697-8b29-4ff3d51718ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:05.716999 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8c3d319c-d7cd-4697-8b29-4ff3d51718ba None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:05.718360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8c3d319c-d7cd-4697-8b29-4ff3d51718ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:05.863579 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1205/2424] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:24:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:24:05.919852 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9eecb7d-c095-4519-acf5-a4bbd4751a7e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:05.920426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9eecb7d-c095-4519-acf5-a4bbd4751a7e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:05.920426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9eecb7d-c095-4519-acf5-a4bbd4751a7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:05.944066 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-d9eecb7d-c095-4519-acf5-a4bbd4751a7e None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:05.945582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d9eecb7d-c095-4519-acf5-a4bbd4751a7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:05.976039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3d319c-d7cd-4697-8b29-4ff3d51718ba None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:05.984049 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1220/2425] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:24:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:24:05.988236 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3d319c-d7cd-4697-8b29-4ff3d51718ba None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:05.994716 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fc513a-fb88-4370-a6ab-48c96fb479dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:05.996126 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fc513a-fb88-4370-a6ab-48c96fb479dc admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:05.996413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fc513a-fb88-4370-a6ab-48c96fb479dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:05.996687 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fc513a-fb88-4370-a6ab-48c96fb479dc admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:05.997344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fc513a-fb88-4370-a6ab-48c96fb479dc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:24:05.997646 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fc513a-fb88-4370-a6ab-48c96fb479dc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:05.999797 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fc513a-fb88-4370-a6ab-48c96fb479dc admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:06.021519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9eecb7d-c095-4519-acf5-a4bbd4751a7e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:06.029255 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1206/2426] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:24:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:24:06.037345 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9eecb7d-c095-4519-acf5-a4bbd4751a7e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:06.045644 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189248f6-5034-4ffc-93ac-dd0ca3115a73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:06.046741 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189248f6-5034-4ffc-93ac-dd0ca3115a73 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:06.046903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189248f6-5034-4ffc-93ac-dd0ca3115a73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:06.047208 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189248f6-5034-4ffc-93ac-dd0ca3115a73 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:06.048517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189248f6-5034-4ffc-93ac-dd0ca3115a73 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:06.050746 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189248f6-5034-4ffc-93ac-dd0ca3115a73 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:06.074002 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb804dc8-18cc-45a0-a8de-82a6aad2ec47 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:06.074500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb804dc8-18cc-45a0-a8de-82a6aad2ec47 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:06.074896 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb804dc8-18cc-45a0-a8de-82a6aad2ec47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:06.102640 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-eb804dc8-18cc-45a0-a8de-82a6aad2ec47 None None] MFA Rules not processed for user `e28a0ba45389417ba8ae92376dba87ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:06.104056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-eb804dc8-18cc-45a0-a8de-82a6aad2ec47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:06.104643 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1221/2427] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:24:06 2022] POST /identity/v3/projects => generated 373 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:24:06.106290 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189248f6-5034-4ffc-93ac-dd0ca3115a73 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:06.178812 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1207/2428] 10.222.0.15 () {58 vars in 1009 bytes} [Tue Sep 13 01:24:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:24:06.194471 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189248f6-5034-4ffc-93ac-dd0ca3115a73 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:06.203797 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9986c411-76c6-4267-91aa-2e8f197f4738 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:06.204624 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9986c411-76c6-4267-91aa-2e8f197f4738 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:06.205196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9986c411-76c6-4267-91aa-2e8f197f4738 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:06.205629 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9986c411-76c6-4267-91aa-2e8f197f4738 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:06.206499 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9986c411-76c6-4267-91aa-2e8f197f4738 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:06.208813 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9986c411-76c6-4267-91aa-2e8f197f4738 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:06.244917 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1222/2429] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:24:06 2022] POST /identity/v3/users => generated 287 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:24:06.249699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb804dc8-18cc-45a0-a8de-82a6aad2ec47 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:06.253306 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9986c411-76c6-4267-91aa-2e8f197f4738 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:06.255123 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9986c411-76c6-4267-91aa-2e8f197f4738 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:06.272651 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb804dc8-18cc-45a0-a8de-82a6aad2ec47 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:06.281127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343df593-3da3-4dc2-85e2-e83b5d7a4b9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:06.282004 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343df593-3da3-4dc2-85e2-e83b5d7a4b9b admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:06.282733 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343df593-3da3-4dc2-85e2-e83b5d7a4b9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:06.283213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343df593-3da3-4dc2-85e2-e83b5d7a4b9b admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:06.284046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343df593-3da3-4dc2-85e2-e83b5d7a4b9b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:06.286153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343df593-3da3-4dc2-85e2-e83b5d7a4b9b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:06.298942 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1208/2430] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:24:06 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:24:06.301352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343df593-3da3-4dc2-85e2-e83b5d7a4b9b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:06.322300 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343df593-3da3-4dc2-85e2-e83b5d7a4b9b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:06.343947 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbacf61-3140-4d7d-9809-519eaca79c50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:06.345545 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbacf61-3140-4d7d-9809-519eaca79c50 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:06.345813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbacf61-3140-4d7d-9809-519eaca79c50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:06.346053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbacf61-3140-4d7d-9809-519eaca79c50 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:06.346747 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbacf61-3140-4d7d-9809-519eaca79c50 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:06.348544 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbacf61-3140-4d7d-9809-519eaca79c50 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:06.349586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:06.391066 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9986c411-76c6-4267-91aa-2e8f197f4738 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:06.397020 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f17038-10fb-4faf-af02-84df45372a8c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d7d8ed4ede4ddca192fce0a497689d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:06.397662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f17038-10fb-4faf-af02-84df45372a8c service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:06.397982 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f17038-10fb-4faf-af02-84df45372a8c service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:06.398216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f17038-10fb-4faf-af02-84df45372a8c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:06.400143 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1209/2431] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:24:06 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:24:06.401111 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbacf61-3140-4d7d-9809-519eaca79c50 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:06.418971 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f17038-10fb-4faf-af02-84df45372a8c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:06.420574 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f17038-10fb-4faf-af02-84df45372a8c service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:06.426574 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbacf61-3140-4d7d-9809-519eaca79c50 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:06.440029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48467794-8cdd-4ee2-887f-f4427ad0c7f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:06.440984 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48467794-8cdd-4ee2-887f-f4427ad0c7f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:06.440984 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48467794-8cdd-4ee2-887f-f4427ad0c7f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:06.441352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48467794-8cdd-4ee2-887f-f4427ad0c7f3 admin admin] PATH_INFO: `/v3/projects/2e2621e390f744039d2731cedf974894/users/cd5712c2ddca43c6a59a26ec93c14f45/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:06.442356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1223/2432] 10.222.0.15 () {60 vars in 1384 bytes} [Tue Sep 13 01:24:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 01:24:06.493137 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48467794-8cdd-4ee2-887f-f4427ad0c7f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e2621e390f744039d2731cedf974894, user_id=cd5712c2ddca43c6a59a26ec93c14f45, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:06.497964 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48467794-8cdd-4ee2-887f-f4427ad0c7f3 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:06.525795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1210/2433] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:24:06 2022] PUT /identity/v3/projects/2e2621e390f744039d2731cedf974894/users/cd5712c2ddca43c6a59a26ec93c14f45/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:24:06.537419 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459d4429-6732-43d1-add2-bdc603863eca None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:06.537607 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459d4429-6732-43d1-add2-bdc603863eca None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:06.537886 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459d4429-6732-43d1-add2-bdc603863eca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:06.603303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-459d4429-6732-43d1-add2-bdc603863eca None None] MFA Rules not processed for user `cd5712c2ddca43c6a59a26ec93c14f45`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:06.604915 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-459d4429-6732-43d1-add2-bdc603863eca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:06.698115 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1224/2434] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:24:06 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:24:06.709661 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48467794-8cdd-4ee2-887f-f4427ad0c7f3 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:06.711736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-48467794-8cdd-4ee2-887f-f4427ad0c7f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:06.732589 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:DELETE=2,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:06.785810 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48467794-8cdd-4ee2-887f-f4427ad0c7f3 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:06.791262 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2988905-9e2a-4136-88c9-01d7acb5b930 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:06.791999 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2988905-9e2a-4136-88c9-01d7acb5b930 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:06.792346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2988905-9e2a-4136-88c9-01d7acb5b930 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:06.792696 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2988905-9e2a-4136-88c9-01d7acb5b930 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:06.804905 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2988905-9e2a-4136-88c9-01d7acb5b930 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:06.806210 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2988905-9e2a-4136-88c9-01d7acb5b930 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:06.818606 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1211/2435] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:24:06 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:24:16.695239 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=39 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:16.814213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=19 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:16.985468 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459d4429-6732-43d1-add2-bdc603863eca None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:16.987058 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-459d4429-6732-43d1-add2-bdc603863eca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:17.045780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459d4429-6732-43d1-add2-bdc603863eca None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:17.049342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e67242-fa30-4414-8da9-ddf18569c150 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:17.049951 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e67242-fa30-4414-8da9-ddf18569c150 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:17.050347 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e67242-fa30-4414-8da9-ddf18569c150 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:17.050685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e67242-fa30-4414-8da9-ddf18569c150 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:17.052462 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-46e67242-fa30-4414-8da9-ddf18569c150 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:17.065461 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e67242-fa30-4414-8da9-ddf18569c150 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:17.066914 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e67242-fa30-4414-8da9-ddf18569c150 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:17.075194 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1225/2436] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:24:16 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:24:19.303543 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efd8147-95cd-4112-842b-2451f7fc8dc7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:19.304077 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efd8147-95cd-4112-842b-2451f7fc8dc7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:19.304077 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efd8147-95cd-4112-842b-2451f7fc8dc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:19.324974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-4efd8147-95cd-4112-842b-2451f7fc8dc7 None None] MFA Rules not processed for user `776733af5b6a4115b2a11d67666a379d`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:19.326252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4efd8147-95cd-4112-842b-2451f7fc8dc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:19.381041 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1212/2437] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:24:19 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:24:19.686779 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e67242-fa30-4414-8da9-ddf18569c150 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:19.697314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e67242-fa30-4414-8da9-ddf18569c150 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:19.702543 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74361f1d-e43f-4696-b330-4a034c557e23 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:19.703291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74361f1d-e43f-4696-b330-4a034c557e23 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:19.703572 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74361f1d-e43f-4696-b330-4a034c557e23 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:19.703846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74361f1d-e43f-4696-b330-4a034c557e23 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:19.706238 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-74361f1d-e43f-4696-b330-4a034c557e23 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:19.717200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74361f1d-e43f-4696-b330-4a034c557e23 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:19.718625 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74361f1d-e43f-4696-b330-4a034c557e23 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:19.729693 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1226/2438] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:24:19 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:24:20.067437 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efd8147-95cd-4112-842b-2451f7fc8dc7 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:20.068836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4efd8147-95cd-4112-842b-2451f7fc8dc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:20.143246 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efd8147-95cd-4112-842b-2451f7fc8dc7 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:20.148177 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01965c3c-ee9f-4e33-893a-d4d7a0d2f953 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8eaff1b843a145a49994dcf089f44315', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:20.148781 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01965c3c-ee9f-4e33-893a-d4d7a0d2f953 service placement] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:20.149059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01965c3c-ee9f-4e33-893a-d4d7a0d2f953 service placement] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:20.149329 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01965c3c-ee9f-4e33-893a-d4d7a0d2f953 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:20.151625 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-01965c3c-ee9f-4e33-893a-d4d7a0d2f953 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:20.164300 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01965c3c-ee9f-4e33-893a-d4d7a0d2f953 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:20.165886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01965c3c-ee9f-4e33-893a-d4d7a0d2f953 service placement] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:20.175684 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1213/2439] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:24:20 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:24:21.813131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74361f1d-e43f-4696-b330-4a034c557e23 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:21.814706 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-74361f1d-e43f-4696-b330-4a034c557e23 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:21.827357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74361f1d-e43f-4696-b330-4a034c557e23 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:21.831140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35683f55-b961-471f-a530-2ad0a0afdd26 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:21.831771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35683f55-b961-471f-a530-2ad0a0afdd26 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:21.832007 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35683f55-b961-471f-a530-2ad0a0afdd26 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:21.832280 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35683f55-b961-471f-a530-2ad0a0afdd26 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:21.833964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-35683f55-b961-471f-a530-2ad0a0afdd26 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:21.844755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35683f55-b961-471f-a530-2ad0a0afdd26 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:21.846174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35683f55-b961-471f-a530-2ad0a0afdd26 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:21.854266 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1227/2440] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:24:21 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:24:25.245322 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c93777b-d18b-419c-ba85-bcb9442033ed None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:25.245847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c93777b-d18b-419c-ba85-bcb9442033ed None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:25.245847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c93777b-d18b-419c-ba85-bcb9442033ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:25.265936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-3c93777b-d18b-419c-ba85-bcb9442033ed None None] MFA Rules not processed for user `e28a0ba45389417ba8ae92376dba87ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:25.267260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3c93777b-d18b-419c-ba85-bcb9442033ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:25.298801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35683f55-b961-471f-a530-2ad0a0afdd26 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:25.300643 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-35683f55-b961-471f-a530-2ad0a0afdd26 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:25.352018 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1214/2441] 10.222.0.15 () {58 vars in 1009 bytes} [Tue Sep 13 01:24:25 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:24:25.367528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c93777b-d18b-419c-ba85-bcb9442033ed None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:25.416581 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35683f55-b961-471f-a530-2ad0a0afdd26 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:25.422608 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:25.424303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f2d1a9-3a8e-44e8-ac6e-d889cbfdfbf0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:25.425101 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f2d1a9-3a8e-44e8-ac6e-d889cbfdfbf0 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:25.425563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f2d1a9-3a8e-44e8-ac6e-d889cbfdfbf0 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:25.425832 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f2d1a9-3a8e-44e8-ac6e-d889cbfdfbf0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:25.427865 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-99f2d1a9-3a8e-44e8-ac6e-d889cbfdfbf0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:25.441516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f2d1a9-3a8e-44e8-ac6e-d889cbfdfbf0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:25.443166 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f2d1a9-3a8e-44e8-ac6e-d889cbfdfbf0 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:25.453064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c93777b-d18b-419c-ba85-bcb9442033ed None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:25.455275 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1228/2442] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:24:25 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:24:25.459214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f2002a-2dda-4284-a54c-554170901409 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d7d8ed4ede4ddca192fce0a497689d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:25.460026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f2002a-2dda-4284-a54c-554170901409 service swift] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:25.460347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f2002a-2dda-4284-a54c-554170901409 service swift] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:25.460717 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f2002a-2dda-4284-a54c-554170901409 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:25.471359 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f2002a-2dda-4284-a54c-554170901409 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:25.473153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f2002a-2dda-4284-a54c-554170901409 service swift] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:25.485785 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1215/2443] 10.222.0.15 () {60 vars in 1384 bytes} [Tue Sep 13 01:24:25 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 01:24:27.042796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f2d1a9-3a8e-44e8-ac6e-d889cbfdfbf0 service nova] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:27.127202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f2d1a9-3a8e-44e8-ac6e-d889cbfdfbf0 service nova] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:27.132364 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743bd1b0-3aaf-491b-b520-1cacef88b328 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:27.133006 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743bd1b0-3aaf-491b-b520-1cacef88b328 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:27.133283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743bd1b0-3aaf-491b-b520-1cacef88b328 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:27.133546 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743bd1b0-3aaf-491b-b520-1cacef88b328 admin admin] PATH_INFO: `/v3/users/cd5712c2ddca43c6a59a26ec93c14f45` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:27.137334 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-743bd1b0-3aaf-491b-b520-1cacef88b328 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cd5712c2ddca43c6a59a26ec93c14f45)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:27.138763 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-743bd1b0-3aaf-491b-b520-1cacef88b328 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:27.141502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:27.207056 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-743bd1b0-3aaf-491b-b520-1cacef88b328 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cd5712c2ddca43c6a59a26ec93c14f45'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:27.207998 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-743bd1b0-3aaf-491b-b520-1cacef88b328 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cd5712c2ddca43c6a59a26ec93c14f45'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:27.218136 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-743bd1b0-3aaf-491b-b520-1cacef88b328 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cd5712c2ddca43c6a59a26ec93c14f45'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:27.219563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-743bd1b0-3aaf-491b-b520-1cacef88b328 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cd5712c2ddca43c6a59a26ec93c14f45'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:27.234818 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-743bd1b0-3aaf-491b-b520-1cacef88b328 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cd5712c2ddca43c6a59a26ec93c14f45'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:27.249739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1229/2444] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:24:27 2022] DELETE /identity/v3/users/cd5712c2ddca43c6a59a26ec93c14f45 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:24:27.254366 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f9399a-0107-476e-9d74-f708dd9cb9e6 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:27.254978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f9399a-0107-476e-9d74-f708dd9cb9e6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:27.254978 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f9399a-0107-476e-9d74-f708dd9cb9e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:27.277174 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-a7f9399a-0107-476e-9d74-f708dd9cb9e6 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:27.278486 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a7f9399a-0107-476e-9d74-f708dd9cb9e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:27.343344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1216/2445] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:24:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:24:27.353930 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743bd1b0-3aaf-491b-b520-1cacef88b328 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:27.355466 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-743bd1b0-3aaf-491b-b520-1cacef88b328 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:27.416901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743bd1b0-3aaf-491b-b520-1cacef88b328 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:27.420824 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7557d9a-1b1a-4235-aa8c-839c24369795 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:27.421602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7557d9a-1b1a-4235-aa8c-839c24369795 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:27.421885 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7557d9a-1b1a-4235-aa8c-839c24369795 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:27.422157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7557d9a-1b1a-4235-aa8c-839c24369795 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:27.432954 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7557d9a-1b1a-4235-aa8c-839c24369795 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:27.434401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7557d9a-1b1a-4235-aa8c-839c24369795 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:27.444486 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1230/2446] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:24:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:24:27.720380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f9399a-0107-476e-9d74-f708dd9cb9e6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:27.722075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a7f9399a-0107-476e-9d74-f708dd9cb9e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:27.736127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f9399a-0107-476e-9d74-f708dd9cb9e6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:27.741640 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de999d01-de60-4422-b64d-147554464091 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:27.742455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de999d01-de60-4422-b64d-147554464091 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:27.742802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de999d01-de60-4422-b64d-147554464091 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:27.743168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de999d01-de60-4422-b64d-147554464091 admin admin] PATH_INFO: `/v3/projects/2e2621e390f744039d2731cedf974894` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:27.744601 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de999d01-de60-4422-b64d-147554464091 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2e2621e390f744039d2731cedf974894)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:27.746806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de999d01-de60-4422-b64d-147554464091 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:27.801736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-de999d01-de60-4422-b64d-147554464091 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2e2621e390f744039d2731cedf974894', 'request_id': 'req-de999d01-de60-4422-b64d-147554464091'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:27.807681 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-de999d01-de60-4422-b64d-147554464091 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2e2621e390f744039d2731cedf974894', 'request_id': 'req-de999d01-de60-4422-b64d-147554464091'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:27.814803 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-de999d01-de60-4422-b64d-147554464091 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2e2621e390f744039d2731cedf974894', 'request_id': 'req-de999d01-de60-4422-b64d-147554464091'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:27.828098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-de999d01-de60-4422-b64d-147554464091 admin admin] The token cache is being invalidate because project 2e2621e390f744039d2731cedf974894 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:24:27.828484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-de999d01-de60-4422-b64d-147554464091 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:27.829179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-de999d01-de60-4422-b64d-147554464091 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:24:27.830775 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1217/2447] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:24:27 2022] DELETE /identity/v3/projects/2e2621e390f744039d2731cedf974894 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:24:27.835764 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee4c4d7-a033-4e83-ae30-655eb6fe296a None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:27.836054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee4c4d7-a033-4e83-ae30-655eb6fe296a None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:27.836326 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee4c4d7-a033-4e83-ae30-655eb6fe296a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:27.857156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5ee4c4d7-a033-4e83-ae30-655eb6fe296a None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:27.858357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5ee4c4d7-a033-4e83-ae30-655eb6fe296a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:27.926890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1231/2448] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:24:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:24:27.933160 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b1e481-bcb5-46f7-8952-e3852824069d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:27.933480 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b1e481-bcb5-46f7-8952-e3852824069d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:27.933708 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b1e481-bcb5-46f7-8952-e3852824069d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:27.953773 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-59b1e481-bcb5-46f7-8952-e3852824069d None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:27.955016 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-59b1e481-bcb5-46f7-8952-e3852824069d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:27.963788 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1218/2449] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:24:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:24:27.968572 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee4c4d7-a033-4e83-ae30-655eb6fe296a None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:27.979074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee4c4d7-a033-4e83-ae30-655eb6fe296a None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:27.983662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f3376a-4da8-47c0-991d-dd01704613cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:27.984269 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f3376a-4da8-47c0-991d-dd01704613cf admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:27.984551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f3376a-4da8-47c0-991d-dd01704613cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:27.984820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f3376a-4da8-47c0-991d-dd01704613cf admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:27.985470 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f3376a-4da8-47c0-991d-dd01704613cf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:24:27.985773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f3376a-4da8-47c0-991d-dd01704613cf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:27.987397 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f3376a-4da8-47c0-991d-dd01704613cf admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:28.001970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1232/2450] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:24:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:24:28.006182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b1e481-bcb5-46f7-8952-e3852824069d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:28.015959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b1e481-bcb5-46f7-8952-e3852824069d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:28.020730 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327fe2b8-330e-4309-b560-043b06be6050 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:28.021341 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327fe2b8-330e-4309-b560-043b06be6050 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:28.021619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327fe2b8-330e-4309-b560-043b06be6050 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:28.021890 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327fe2b8-330e-4309-b560-043b06be6050 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:28.022528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327fe2b8-330e-4309-b560-043b06be6050 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:28.024755 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327fe2b8-330e-4309-b560-043b06be6050 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:28.054695 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1219/2451] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:24:28 2022] POST /identity/v3/projects => generated 371 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:24:28.059347 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f3376a-4da8-47c0-991d-dd01704613cf admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:28.069739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f3376a-4da8-47c0-991d-dd01704613cf admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:28.078379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:28.135465 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b22280b-c8f3-46d1-ac29-cd74ae78e57d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:28.136090 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b22280b-c8f3-46d1-ac29-cd74ae78e57d admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:28.136370 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b22280b-c8f3-46d1-ac29-cd74ae78e57d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:28.136638 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b22280b-c8f3-46d1-ac29-cd74ae78e57d admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:28.137267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b22280b-c8f3-46d1-ac29-cd74ae78e57d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:28.139417 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b22280b-c8f3-46d1-ac29-cd74ae78e57d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:28.161316 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1233/2452] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:24:28 2022] POST /identity/v3/users => generated 286 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:24:28.166615 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327fe2b8-330e-4309-b560-043b06be6050 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:28.177215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327fe2b8-330e-4309-b560-043b06be6050 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:28.182616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1caa42-0c23-4c68-85e5-1852046f99b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:28.183335 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1caa42-0c23-4c68-85e5-1852046f99b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:28.183675 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1caa42-0c23-4c68-85e5-1852046f99b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:28.184007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1caa42-0c23-4c68-85e5-1852046f99b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:28.184736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1caa42-0c23-4c68-85e5-1852046f99b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:28.186417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1caa42-0c23-4c68-85e5-1852046f99b9 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:28.197886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1220/2453] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:24:28 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:24:28.202195 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b22280b-c8f3-46d1-ac29-cd74ae78e57d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:28.217442 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b22280b-c8f3-46d1-ac29-cd74ae78e57d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:28.222537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec03fd0-f06d-45f4-8302-083c7292cc35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:28.223252 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec03fd0-f06d-45f4-8302-083c7292cc35 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:28.223522 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec03fd0-f06d-45f4-8302-083c7292cc35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:28.223792 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec03fd0-f06d-45f4-8302-083c7292cc35 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:28.224430 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec03fd0-f06d-45f4-8302-083c7292cc35 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:28.226050 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec03fd0-f06d-45f4-8302-083c7292cc35 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:28.238030 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1234/2454] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:24:28 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:24:28.242066 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1caa42-0c23-4c68-85e5-1852046f99b9 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:28.253309 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1caa42-0c23-4c68-85e5-1852046f99b9 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:28.258169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca7ba09-96f5-445a-9550-cd6514ee593b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:28.258969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca7ba09-96f5-445a-9550-cd6514ee593b admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:28.259310 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca7ba09-96f5-445a-9550-cd6514ee593b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:28.259714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca7ba09-96f5-445a-9550-cd6514ee593b admin admin] PATH_INFO: `/v3/projects/99cfba5cf9d54e10ba5fc677bf713e2f/users/38ba0a3f69474e4ba60361b5c48f4579/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:28.271864 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:28.279549 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca7ba09-96f5-445a-9550-cd6514ee593b admin admin] RBAC: Authorizing `identity:create_grant(project_id=99cfba5cf9d54e10ba5fc677bf713e2f, user_id=38ba0a3f69474e4ba60361b5c48f4579, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:28.281914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca7ba09-96f5-445a-9550-cd6514ee593b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:28.299650 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1221/2455] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:24:28 2022] PUT /identity/v3/projects/99cfba5cf9d54e10ba5fc677bf713e2f/users/38ba0a3f69474e4ba60361b5c48f4579/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:24:28.302816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68c12f0-ee72-482d-b62b-6fc2d06276b5 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:28.303209 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68c12f0-ee72-482d-b62b-6fc2d06276b5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:28.303544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68c12f0-ee72-482d-b62b-6fc2d06276b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:28.328948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a68c12f0-ee72-482d-b62b-6fc2d06276b5 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:28.330291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a68c12f0-ee72-482d-b62b-6fc2d06276b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:28.452239 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1235/2456] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:24:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:24:28.464439 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca7ba09-96f5-445a-9550-cd6514ee593b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:28.467035 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cca7ba09-96f5-445a-9550-cd6514ee593b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:28.543872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca7ba09-96f5-445a-9550-cd6514ee593b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:28.549517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043ff375-8552-4007-ac89-bd91a3a77802 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:28.550481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043ff375-8552-4007-ac89-bd91a3a77802 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:28.550841 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043ff375-8552-4007-ac89-bd91a3a77802 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:28.551141 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043ff375-8552-4007-ac89-bd91a3a77802 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:28.561636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043ff375-8552-4007-ac89-bd91a3a77802 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:28.563122 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043ff375-8552-4007-ac89-bd91a3a77802 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:28.573186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1222/2457] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:24:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:24:29.253260 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181b9233-2056-4aa4-9de5-eb0eaa6305e9 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:29.253781 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181b9233-2056-4aa4-9de5-eb0eaa6305e9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:29.254079 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181b9233-2056-4aa4-9de5-eb0eaa6305e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:29.294560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-181b9233-2056-4aa4-9de5-eb0eaa6305e9 None None] MFA Rules not processed for user `38ba0a3f69474e4ba60361b5c48f4579`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:29.295814 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-181b9233-2056-4aa4-9de5-eb0eaa6305e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:29.381904 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1236/2458] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:24:29 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:24:29.396401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043ff375-8552-4007-ac89-bd91a3a77802 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:29.397815 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-043ff375-8552-4007-ac89-bd91a3a77802 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:29.529245 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043ff375-8552-4007-ac89-bd91a3a77802 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:29.535081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c78e99f-fa0e-48f3-9dd8-068b65122f9d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:29.535950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c78e99f-fa0e-48f3-9dd8-068b65122f9d service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:29.536252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c78e99f-fa0e-48f3-9dd8-068b65122f9d service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:29.536522 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c78e99f-fa0e-48f3-9dd8-068b65122f9d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:29.546417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c78e99f-fa0e-48f3-9dd8-068b65122f9d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:29.547896 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c78e99f-fa0e-48f3-9dd8-068b65122f9d service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:29.559105 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1223/2459] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:24:29 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:24:29.953358 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181b9233-2056-4aa4-9de5-eb0eaa6305e9 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:29.956987 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-181b9233-2056-4aa4-9de5-eb0eaa6305e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:30.009576 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:30.077216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181b9233-2056-4aa4-9de5-eb0eaa6305e9 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:30.086355 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f563045c-1763-4178-9a0a-e44269650b82 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:30.087033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f563045c-1763-4178-9a0a-e44269650b82 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:30.087308 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f563045c-1763-4178-9a0a-e44269650b82 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:30.088131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f563045c-1763-4178-9a0a-e44269650b82 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:30.090174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f563045c-1763-4178-9a0a-e44269650b82 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:30.149407 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f563045c-1763-4178-9a0a-e44269650b82 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:30.151553 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f563045c-1763-4178-9a0a-e44269650b82 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:30.161721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1237/2460] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:24:29 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 209 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:24:31.025522 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495a342c-54e8-4032-9dbb-be1d341e545e None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:31.026082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495a342c-54e8-4032-9dbb-be1d341e545e None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:31.026082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495a342c-54e8-4032-9dbb-be1d341e545e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:31.045363 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-495a342c-54e8-4032-9dbb-be1d341e545e None None] MFA Rules not processed for user `38ba0a3f69474e4ba60361b5c48f4579`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:31.046585 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-495a342c-54e8-4032-9dbb-be1d341e545e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:31.053619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1224/2461] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:24:31 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:24:40.156950 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=47 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:40.252310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f563045c-1763-4178-9a0a-e44269650b82 service nova] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:40.254924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f563045c-1763-4178-9a0a-e44269650b82 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:40.271341 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f563045c-1763-4178-9a0a-e44269650b82 service nova] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:40.278307 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f60432-b19e-4a15-9db9-354c0792c646 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:40.278964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f60432-b19e-4a15-9db9-354c0792c646 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:40.279246 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f60432-b19e-4a15-9db9-354c0792c646 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:40.279522 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f60432-b19e-4a15-9db9-354c0792c646 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:40.281688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c5f60432-b19e-4a15-9db9-354c0792c646 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:40.299240 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f60432-b19e-4a15-9db9-354c0792c646 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:40.300836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f60432-b19e-4a15-9db9-354c0792c646 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:40.314924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1238/2462] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:24:40 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:24:41.039853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=59,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:24:44.132832 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495a342c-54e8-4032-9dbb-be1d341e545e None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:44.153761 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495a342c-54e8-4032-9dbb-be1d341e545e None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:44.163021 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1445093d-eb69-4322-949a-224e5e16b906 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:44.164236 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1445093d-eb69-4322-949a-224e5e16b906 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:44.167098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1445093d-eb69-4322-949a-224e5e16b906 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:44.167743 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1445093d-eb69-4322-949a-224e5e16b906 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:44.170465 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1445093d-eb69-4322-949a-224e5e16b906 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:44.196118 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1445093d-eb69-4322-949a-224e5e16b906 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:44.198945 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1445093d-eb69-4322-949a-224e5e16b906 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:44.227343 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1225/2463] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:24:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:24:45.182170 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46591d93-ca18-468c-85ee-e49093859d62 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:45.182662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46591d93-ca18-468c-85ee-e49093859d62 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:45.182755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46591d93-ca18-468c-85ee-e49093859d62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:45.203453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-46591d93-ca18-468c-85ee-e49093859d62 None None] MFA Rules not processed for user `e28a0ba45389417ba8ae92376dba87ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:45.204337 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-46591d93-ca18-468c-85ee-e49093859d62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:45.260817 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1239/2464] 10.222.0.15 () {58 vars in 967 bytes} [Tue Sep 13 01:24:45 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:24:45.273144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1445093d-eb69-4322-949a-224e5e16b906 service nova] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:24:45.274619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1445093d-eb69-4322-949a-224e5e16b906 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:45.357745 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1445093d-eb69-4322-949a-224e5e16b906 service nova] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:24:45.363656 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b512ac0c-243d-4571-bd9f-77f92ecf3967 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d7d8ed4ede4ddca192fce0a497689d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:24:45.364287 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b512ac0c-243d-4571-bd9f-77f92ecf3967 service swift] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:45.364568 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b512ac0c-243d-4571-bd9f-77f92ecf3967 service swift] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:45.364843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b512ac0c-243d-4571-bd9f-77f92ecf3967 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:45.377209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b512ac0c-243d-4571-bd9f-77f92ecf3967 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:24:45.379287 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b512ac0c-243d-4571-bd9f-77f92ecf3967 service swift] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:24:45.397551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1226/2465] 10.222.0.15 () {60 vars in 1384 bytes} [Tue Sep 13 01:24:45 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 01:24:50.874302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af288eb8-633c-403b-93ab-d067f3bc3d5d None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:24:50.874769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af288eb8-633c-403b-93ab-d067f3bc3d5d None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:24:50.874911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af288eb8-633c-403b-93ab-d067f3bc3d5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:24:50.893969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-af288eb8-633c-403b-93ab-d067f3bc3d5d None None] MFA Rules not processed for user `38ba0a3f69474e4ba60361b5c48f4579`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:24:50.895188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-af288eb8-633c-403b-93ab-d067f3bc3d5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:24:50.902598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1240/2466] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:24:50 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:24:55.387782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=38 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:00.888593 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=34 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:08.926585 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b512ac0c-243d-4571-bd9f-77f92ecf3967 service swift] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:08.936958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b512ac0c-243d-4571-bd9f-77f92ecf3967 service swift] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:08.940521 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5934fc-0312-4edc-b257-f8f7e2e22cf3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:08.941110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5934fc-0312-4edc-b257-f8f7e2e22cf3 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:08.941387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5934fc-0312-4edc-b257-f8f7e2e22cf3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:08.941657 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5934fc-0312-4edc-b257-f8f7e2e22cf3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:08.943397 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5a5934fc-0312-4edc-b257-f8f7e2e22cf3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:08.956402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5934fc-0312-4edc-b257-f8f7e2e22cf3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:08.957848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5934fc-0312-4edc-b257-f8f7e2e22cf3 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:08.968918 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1227/2467] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:12.036359 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af288eb8-633c-403b-93ab-d067f3bc3d5d None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:12.037822 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-af288eb8-633c-403b-93ab-d067f3bc3d5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:12.112464 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5934fc-0312-4edc-b257-f8f7e2e22cf3 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:12.114455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5a5934fc-0312-4edc-b257-f8f7e2e22cf3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:12.132511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5934fc-0312-4edc-b257-f8f7e2e22cf3 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:12.138560 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68602eab-7f27-4475-a84c-ebc9cfd16b54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:12.139954 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68602eab-7f27-4475-a84c-ebc9cfd16b54 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:12.140416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68602eab-7f27-4475-a84c-ebc9cfd16b54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:12.140862 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68602eab-7f27-4475-a84c-ebc9cfd16b54 admin admin] PATH_INFO: `/v3/users/349e19300f1e48c3ac44fc34611c7f4a` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:12.142420 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68602eab-7f27-4475-a84c-ebc9cfd16b54 admin admin] RBAC: Authorizing `identity:delete_user(user_id=349e19300f1e48c3ac44fc34611c7f4a)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:12.145026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68602eab-7f27-4475-a84c-ebc9cfd16b54 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:12.177485 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af288eb8-633c-403b-93ab-d067f3bc3d5d None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:12.184829 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d877cea-1248-4e26-9a32-a3dc2fa63348 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8eaff1b843a145a49994dcf089f44315', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:12.185550 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d877cea-1248-4e26-9a32-a3dc2fa63348 service placement] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:12.185896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d877cea-1248-4e26-9a32-a3dc2fa63348 service placement] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:12.186233 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d877cea-1248-4e26-9a32-a3dc2fa63348 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:12.188516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1d877cea-1248-4e26-9a32-a3dc2fa63348 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:12.210389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d877cea-1248-4e26-9a32-a3dc2fa63348 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:12.212645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d877cea-1248-4e26-9a32-a3dc2fa63348 service placement] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:12.224157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-68602eab-7f27-4475-a84c-ebc9cfd16b54 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '349e19300f1e48c3ac44fc34611c7f4a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:12.224807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1241/2468] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:12 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:12.224928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-68602eab-7f27-4475-a84c-ebc9cfd16b54 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '349e19300f1e48c3ac44fc34611c7f4a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:12.234617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-68602eab-7f27-4475-a84c-ebc9cfd16b54 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '349e19300f1e48c3ac44fc34611c7f4a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:12.235702 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-68602eab-7f27-4475-a84c-ebc9cfd16b54 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '349e19300f1e48c3ac44fc34611c7f4a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:12.244217 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-68602eab-7f27-4475-a84c-ebc9cfd16b54 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '349e19300f1e48c3ac44fc34611c7f4a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:12.263768 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1228/2469] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:25:12 2022] DELETE /identity/v3/users/349e19300f1e48c3ac44fc34611c7f4a => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:25:12.440124 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d877cea-1248-4e26-9a32-a3dc2fa63348 service placement] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:12.452125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d877cea-1248-4e26-9a32-a3dc2fa63348 service placement] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:12.515419 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50cb544-dac9-41d7-a081-d195b7010d71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:12.516075 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50cb544-dac9-41d7-a081-d195b7010d71 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:12.516353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50cb544-dac9-41d7-a081-d195b7010d71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:12.516640 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50cb544-dac9-41d7-a081-d195b7010d71 admin admin] PATH_INFO: `/v3/projects/9056ec686513412a87c292387e25e870` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:12.517880 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50cb544-dac9-41d7-a081-d195b7010d71 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9056ec686513412a87c292387e25e870)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:12.519901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50cb544-dac9-41d7-a081-d195b7010d71 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:12.564912 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b50cb544-dac9-41d7-a081-d195b7010d71 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9056ec686513412a87c292387e25e870', 'request_id': 'req-b50cb544-dac9-41d7-a081-d195b7010d71'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:12.570940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b50cb544-dac9-41d7-a081-d195b7010d71 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9056ec686513412a87c292387e25e870', 'request_id': 'req-b50cb544-dac9-41d7-a081-d195b7010d71'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:12.576932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b50cb544-dac9-41d7-a081-d195b7010d71 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9056ec686513412a87c292387e25e870', 'request_id': 'req-b50cb544-dac9-41d7-a081-d195b7010d71'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:12.592599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b50cb544-dac9-41d7-a081-d195b7010d71 admin admin] The token cache is being invalidate because project 9056ec686513412a87c292387e25e870 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:25:12.593146 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b50cb544-dac9-41d7-a081-d195b7010d71 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:12.593844 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b50cb544-dac9-41d7-a081-d195b7010d71 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:12.599292 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1242/2470] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:25:12 2022] DELETE /identity/v3/projects/9056ec686513412a87c292387e25e870 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:12.600967 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a92b7e3-cd4e-4e91-b148-43d6f5fd30e0 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:12.601385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a92b7e3-cd4e-4e91-b148-43d6f5fd30e0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:12.601738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a92b7e3-cd4e-4e91-b148-43d6f5fd30e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:12.621426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7a92b7e3-cd4e-4e91-b148-43d6f5fd30e0 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:12.622546 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7a92b7e3-cd4e-4e91-b148-43d6f5fd30e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:12.681645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1229/2471] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:12.689477 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c6b520-b7dc-4094-b343-0c44eaa663f7 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:12.689787 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c6b520-b7dc-4094-b343-0c44eaa663f7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:12.690072 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c6b520-b7dc-4094-b343-0c44eaa663f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:12.711268 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-18c6b520-b7dc-4094-b343-0c44eaa663f7 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:12.712461 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-18c6b520-b7dc-4094-b343-0c44eaa663f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:12.720296 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1243/2472] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:12.724375 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a92b7e3-cd4e-4e91-b148-43d6f5fd30e0 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:12.734034 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a92b7e3-cd4e-4e91-b148-43d6f5fd30e0 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:12.738635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efac71fd-753d-4397-969d-321064ce5952 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:12.739261 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efac71fd-753d-4397-969d-321064ce5952 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:12.739542 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efac71fd-753d-4397-969d-321064ce5952 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:12.739814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efac71fd-753d-4397-969d-321064ce5952 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:12.740447 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efac71fd-753d-4397-969d-321064ce5952 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:25:12.740751 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efac71fd-753d-4397-969d-321064ce5952 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:12.742825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efac71fd-753d-4397-969d-321064ce5952 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:12.754149 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1230/2473] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:25:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:25:12.758976 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c6b520-b7dc-4094-b343-0c44eaa663f7 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:12.768173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c6b520-b7dc-4094-b343-0c44eaa663f7 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:12.772731 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158f0174-0f72-44cb-bf6f-4ef677d0de0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:12.773339 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158f0174-0f72-44cb-bf6f-4ef677d0de0e admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:12.773616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158f0174-0f72-44cb-bf6f-4ef677d0de0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:12.773877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158f0174-0f72-44cb-bf6f-4ef677d0de0e admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:12.774516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158f0174-0f72-44cb-bf6f-4ef677d0de0e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:12.776395 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158f0174-0f72-44cb-bf6f-4ef677d0de0e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:12.787219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:12.804173 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1244/2474] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:25:12 2022] POST /identity/v3/projects => generated 375 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:12.810343 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efac71fd-753d-4397-969d-321064ce5952 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:12.820186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efac71fd-753d-4397-969d-321064ce5952 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:12.862231 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:12.877175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f45ca7-0a8e-4921-ab59-9a64d6635dea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:12.877776 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f45ca7-0a8e-4921-ab59-9a64d6635dea admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:12.878057 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f45ca7-0a8e-4921-ab59-9a64d6635dea admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:12.878355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f45ca7-0a8e-4921-ab59-9a64d6635dea admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:12.879006 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f45ca7-0a8e-4921-ab59-9a64d6635dea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:12.880903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f45ca7-0a8e-4921-ab59-9a64d6635dea admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:12.903520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1231/2475] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:25:12 2022] POST /identity/v3/users => generated 288 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:12.907661 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158f0174-0f72-44cb-bf6f-4ef677d0de0e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:12.918032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158f0174-0f72-44cb-bf6f-4ef677d0de0e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:12.922727 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfdbdfe6-bfb1-4f61-9186-752138a9d89d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:12.923389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdbdfe6-bfb1-4f61-9186-752138a9d89d admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:12.923671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdbdfe6-bfb1-4f61-9186-752138a9d89d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:12.923945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdbdfe6-bfb1-4f61-9186-752138a9d89d admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:12.924582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfdbdfe6-bfb1-4f61-9186-752138a9d89d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:12.926191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfdbdfe6-bfb1-4f61-9186-752138a9d89d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:12.936872 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1245/2476] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:12.940610 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f45ca7-0a8e-4921-ab59-9a64d6635dea admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:12.951172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f45ca7-0a8e-4921-ab59-9a64d6635dea admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:12.955688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c459340-8142-4fa0-9c84-236c80d5c002 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:12.956405 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c459340-8142-4fa0-9c84-236c80d5c002 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:12.956683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c459340-8142-4fa0-9c84-236c80d5c002 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:12.956951 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c459340-8142-4fa0-9c84-236c80d5c002 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:12.957573 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c459340-8142-4fa0-9c84-236c80d5c002 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:12.959236 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c459340-8142-4fa0-9c84-236c80d5c002 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:12.969556 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1232/2477] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:12 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:12.973463 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfdbdfe6-bfb1-4f61-9186-752138a9d89d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:12.983315 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfdbdfe6-bfb1-4f61-9186-752138a9d89d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:12.987916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d9945b-4cb0-47ed-aa51-234b951783be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:12.988553 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d9945b-4cb0-47ed-aa51-234b951783be admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:12.988831 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d9945b-4cb0-47ed-aa51-234b951783be admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:12.989106 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d9945b-4cb0-47ed-aa51-234b951783be admin admin] PATH_INFO: `/v3/projects/66ec56a460d045a3a3f105b9e72f9e20/users/b1a5d55e95f9421f98df6d99e80f6cd4/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:13.004125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d9945b-4cb0-47ed-aa51-234b951783be admin admin] RBAC: Authorizing `identity:create_grant(project_id=66ec56a460d045a3a3f105b9e72f9e20, user_id=b1a5d55e95f9421f98df6d99e80f6cd4, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:13.006772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d9945b-4cb0-47ed-aa51-234b951783be admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:13.020010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1246/2478] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:12 2022] PUT /identity/v3/projects/66ec56a460d045a3a3f105b9e72f9e20/users/b1a5d55e95f9421f98df6d99e80f6cd4/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:13.030062 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55e4eae-5610-4cce-8029-7d9c18bc6404 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:13.030345 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55e4eae-5610-4cce-8029-7d9c18bc6404 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:13.030614 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55e4eae-5610-4cce-8029-7d9c18bc6404 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:13.058580 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b55e4eae-5610-4cce-8029-7d9c18bc6404 None None] MFA Rules not processed for user `b1a5d55e95f9421f98df6d99e80f6cd4`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:13.059810 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b55e4eae-5610-4cce-8029-7d9c18bc6404 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:13.122008 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1233/2479] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:13 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:13.133208 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d9945b-4cb0-47ed-aa51-234b951783be admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:13.135095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-24d9945b-4cb0-47ed-aa51-234b951783be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:13.225066 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d9945b-4cb0-47ed-aa51-234b951783be admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:13.231904 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f7b5b5-6439-42e1-a4a9-6ab3632dcad3 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:13.232196 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f7b5b5-6439-42e1-a4a9-6ab3632dcad3 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:13.232501 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f7b5b5-6439-42e1-a4a9-6ab3632dcad3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:13.232786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eca4cd7-49c9-4375-9ec8-66890608ef58 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:13.233468 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eca4cd7-49c9-4375-9ec8-66890608ef58 service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:13.233812 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eca4cd7-49c9-4375-9ec8-66890608ef58 service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:13.234048 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eca4cd7-49c9-4375-9ec8-66890608ef58 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:13.243110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eca4cd7-49c9-4375-9ec8-66890608ef58 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:13.244679 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eca4cd7-49c9-4375-9ec8-66890608ef58 service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:13.253236 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-27f7b5b5-6439-42e1-a4a9-6ab3632dcad3 None None] MFA Rules not processed for user `e28a0ba45389417ba8ae92376dba87ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:13.253637 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1247/2480] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:13 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:13.254427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-27f7b5b5-6439-42e1-a4a9-6ab3632dcad3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:13.321813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1234/2481] 10.222.0.15 () {58 vars in 1009 bytes} [Tue Sep 13 01:25:13 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:25:13.344649 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eca4cd7-49c9-4375-9ec8-66890608ef58 service nova] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:13.348708 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8eca4cd7-49c9-4375-9ec8-66890608ef58 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:13.439379 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eca4cd7-49c9-4375-9ec8-66890608ef58 service nova] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:13.444440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae50f23-ddec-4f3c-b24a-463b44bf0d9b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d7d8ed4ede4ddca192fce0a497689d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:13.445043 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae50f23-ddec-4f3c-b24a-463b44bf0d9b service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:13.445354 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae50f23-ddec-4f3c-b24a-463b44bf0d9b service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:13.445626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae50f23-ddec-4f3c-b24a-463b44bf0d9b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:13.455549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae50f23-ddec-4f3c-b24a-463b44bf0d9b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:13.455549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae50f23-ddec-4f3c-b24a-463b44bf0d9b service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:13.466725 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1248/2482] 10.222.0.15 () {60 vars in 1384 bytes} [Tue Sep 13 01:25:13 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 01:25:13.468503 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f7b5b5-6439-42e1-a4a9-6ab3632dcad3 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:13.571176 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f7b5b5-6439-42e1-a4a9-6ab3632dcad3 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:13.580430 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c56f35-852c-4612-918f-980d25d41cb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:13.582130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c56f35-852c-4612-918f-980d25d41cb7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:13.582762 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c56f35-852c-4612-918f-980d25d41cb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:13.583724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c56f35-852c-4612-918f-980d25d41cb7 admin admin] PATH_INFO: `/v3/users/b1a5d55e95f9421f98df6d99e80f6cd4` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:13.586379 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c56f35-852c-4612-918f-980d25d41cb7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b1a5d55e95f9421f98df6d99e80f6cd4)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:13.591845 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c56f35-852c-4612-918f-980d25d41cb7 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:13.595237 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:13.652678 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-54c56f35-852c-4612-918f-980d25d41cb7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b1a5d55e95f9421f98df6d99e80f6cd4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:13.653328 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-54c56f35-852c-4612-918f-980d25d41cb7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b1a5d55e95f9421f98df6d99e80f6cd4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:13.664686 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-54c56f35-852c-4612-918f-980d25d41cb7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b1a5d55e95f9421f98df6d99e80f6cd4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:13.665398 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-54c56f35-852c-4612-918f-980d25d41cb7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b1a5d55e95f9421f98df6d99e80f6cd4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:13.678033 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-54c56f35-852c-4612-918f-980d25d41cb7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b1a5d55e95f9421f98df6d99e80f6cd4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:13.698578 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1235/2483] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:25:13 2022] DELETE /identity/v3/users/b1a5d55e95f9421f98df6d99e80f6cd4 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:25:13.703438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b441ea00-e064-4050-99a5-5498fee17032 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:13.703746 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b441ea00-e064-4050-99a5-5498fee17032 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:13.704045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b441ea00-e064-4050-99a5-5498fee17032 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:13.735825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-b441ea00-e064-4050-99a5-5498fee17032 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:13.737091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b441ea00-e064-4050-99a5-5498fee17032 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:13.782384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:13.826283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1249/2484] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:13.837037 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c56f35-852c-4612-918f-980d25d41cb7 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:13.838561 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-54c56f35-852c-4612-918f-980d25d41cb7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:13.910661 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c56f35-852c-4612-918f-980d25d41cb7 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:13.914782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c72e8f-fab9-4b1b-866f-74fd977b8aff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:13.915518 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c72e8f-fab9-4b1b-866f-74fd977b8aff service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:13.915900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c72e8f-fab9-4b1b-866f-74fd977b8aff service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:13.916452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c72e8f-fab9-4b1b-866f-74fd977b8aff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:13.927673 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c72e8f-fab9-4b1b-866f-74fd977b8aff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:13.929115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c72e8f-fab9-4b1b-866f-74fd977b8aff service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:13.944144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1236/2485] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:14.276607 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b441ea00-e064-4050-99a5-5498fee17032 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:14.278491 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b441ea00-e064-4050-99a5-5498fee17032 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:14.294566 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b441ea00-e064-4050-99a5-5498fee17032 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:14.303043 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01ece6f-7326-4793-9776-4f44c762e77e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:14.304397 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01ece6f-7326-4793-9776-4f44c762e77e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:14.305071 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01ece6f-7326-4793-9776-4f44c762e77e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:14.305412 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01ece6f-7326-4793-9776-4f44c762e77e admin admin] PATH_INFO: `/v3/projects/66ec56a460d045a3a3f105b9e72f9e20` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:14.307414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01ece6f-7326-4793-9776-4f44c762e77e admin admin] RBAC: Authorizing `identity:delete_project(project_id=66ec56a460d045a3a3f105b9e72f9e20)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:14.310785 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01ece6f-7326-4793-9776-4f44c762e77e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:14.400989 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f01ece6f-7326-4793-9776-4f44c762e77e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '66ec56a460d045a3a3f105b9e72f9e20', 'request_id': 'req-f01ece6f-7326-4793-9776-4f44c762e77e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:14.419140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f01ece6f-7326-4793-9776-4f44c762e77e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '66ec56a460d045a3a3f105b9e72f9e20', 'request_id': 'req-f01ece6f-7326-4793-9776-4f44c762e77e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:14.429537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f01ece6f-7326-4793-9776-4f44c762e77e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '66ec56a460d045a3a3f105b9e72f9e20', 'request_id': 'req-f01ece6f-7326-4793-9776-4f44c762e77e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:14.454815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f01ece6f-7326-4793-9776-4f44c762e77e admin admin] The token cache is being invalidate because project 66ec56a460d045a3a3f105b9e72f9e20 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:25:14.455056 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f01ece6f-7326-4793-9776-4f44c762e77e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:14.456776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f01ece6f-7326-4793-9776-4f44c762e77e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:14.460396 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1250/2486] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:25:14 2022] DELETE /identity/v3/projects/66ec56a460d045a3a3f105b9e72f9e20 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:14.465719 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20712377-a04a-4c81-8975-6bdeb62983da None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:14.466289 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20712377-a04a-4c81-8975-6bdeb62983da None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:14.466289 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20712377-a04a-4c81-8975-6bdeb62983da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:14.496415 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-20712377-a04a-4c81-8975-6bdeb62983da None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:14.497724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-20712377-a04a-4c81-8975-6bdeb62983da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:14.575365 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1237/2487] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:14.583651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4c8cd4-c118-47a5-8d4b-6ba454a969b4 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:14.584328 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4c8cd4-c118-47a5-8d4b-6ba454a969b4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:14.584945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4c8cd4-c118-47a5-8d4b-6ba454a969b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:14.613122 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9e4c8cd4-c118-47a5-8d4b-6ba454a969b4 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:14.615257 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9e4c8cd4-c118-47a5-8d4b-6ba454a969b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:14.628287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1251/2488] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:14.629577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20712377-a04a-4c81-8975-6bdeb62983da None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:14.642591 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20712377-a04a-4c81-8975-6bdeb62983da None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:14.647734 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31865473-3261-4255-91c6-9b349ef61798 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:14.648698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31865473-3261-4255-91c6-9b349ef61798 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:14.649020 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31865473-3261-4255-91c6-9b349ef61798 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:14.649293 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31865473-3261-4255-91c6-9b349ef61798 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:14.649975 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31865473-3261-4255-91c6-9b349ef61798 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:25:14.650281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31865473-3261-4255-91c6-9b349ef61798 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:14.651950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31865473-3261-4255-91c6-9b349ef61798 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:14.670533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1238/2489] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:25:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:25:14.672387 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4c8cd4-c118-47a5-8d4b-6ba454a969b4 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:14.684947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4c8cd4-c118-47a5-8d4b-6ba454a969b4 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:14.691296 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2044219b-35ca-4bdc-8917-f0a1378a5f3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:14.692222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2044219b-35ca-4bdc-8917-f0a1378a5f3f admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:14.692580 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2044219b-35ca-4bdc-8917-f0a1378a5f3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:14.693192 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2044219b-35ca-4bdc-8917-f0a1378a5f3f admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:14.694101 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2044219b-35ca-4bdc-8917-f0a1378a5f3f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:14.697144 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2044219b-35ca-4bdc-8917-f0a1378a5f3f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:14.735508 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1252/2490] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:25:14 2022] POST /identity/v3/projects => generated 403 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:14.736802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31865473-3261-4255-91c6-9b349ef61798 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:14.749488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31865473-3261-4255-91c6-9b349ef61798 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:14.755252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:14.817943 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4954a29f-cad8-4589-a0af-4f133ae3b7e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:14.818664 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4954a29f-cad8-4589-a0af-4f133ae3b7e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:14.819061 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4954a29f-cad8-4589-a0af-4f133ae3b7e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:14.819400 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4954a29f-cad8-4589-a0af-4f133ae3b7e6 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:14.820143 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4954a29f-cad8-4589-a0af-4f133ae3b7e6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:14.822170 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4954a29f-cad8-4589-a0af-4f133ae3b7e6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:14.848759 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1239/2491] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:25:14 2022] POST /identity/v3/users => generated 302 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:14.853152 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2044219b-35ca-4bdc-8917-f0a1378a5f3f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:14.863866 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2044219b-35ca-4bdc-8917-f0a1378a5f3f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:14.869356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569fea9d-ead5-4ddd-b8bf-715573dbe0c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:14.869976 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569fea9d-ead5-4ddd-b8bf-715573dbe0c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:14.870254 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569fea9d-ead5-4ddd-b8bf-715573dbe0c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:14.870599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569fea9d-ead5-4ddd-b8bf-715573dbe0c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:14.871272 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569fea9d-ead5-4ddd-b8bf-715573dbe0c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:14.872942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569fea9d-ead5-4ddd-b8bf-715573dbe0c3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:14.887449 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1253/2492] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:14 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:14.888454 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4954a29f-cad8-4589-a0af-4f133ae3b7e6 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:14.901436 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4954a29f-cad8-4589-a0af-4f133ae3b7e6 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:14.906510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25cedec-0738-4cb6-8eb7-05009b7863a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:14.907149 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25cedec-0738-4cb6-8eb7-05009b7863a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:14.907427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25cedec-0738-4cb6-8eb7-05009b7863a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:14.907702 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25cedec-0738-4cb6-8eb7-05009b7863a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:14.908337 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25cedec-0738-4cb6-8eb7-05009b7863a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:14.910075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25cedec-0738-4cb6-8eb7-05009b7863a8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:14.925152 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1240/2493] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:14 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:25:14.926293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569fea9d-ead5-4ddd-b8bf-715573dbe0c3 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:14.940890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569fea9d-ead5-4ddd-b8bf-715573dbe0c3 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:14.947843 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca5c5ad-590d-4b9f-842f-422a1f14fb3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:14.948531 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca5c5ad-590d-4b9f-842f-422a1f14fb3c admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:14.948815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca5c5ad-590d-4b9f-842f-422a1f14fb3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:14.949091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca5c5ad-590d-4b9f-842f-422a1f14fb3c admin admin] PATH_INFO: `/v3/projects/bc5cf717227340829fa38e3159cf8c35/users/7d360f8d42634cb88e17acbb3aaab276/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:14.966506 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca5c5ad-590d-4b9f-842f-422a1f14fb3c admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc5cf717227340829fa38e3159cf8c35, user_id=7d360f8d42634cb88e17acbb3aaab276, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:14.969222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca5c5ad-590d-4b9f-842f-422a1f14fb3c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:14.982900 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1254/2494] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:14 2022] PUT /identity/v3/projects/bc5cf717227340829fa38e3159cf8c35/users/7d360f8d42634cb88e17acbb3aaab276/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:25:14.995462 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9044a2d2-ad08-4a0c-b1d0-526d29f5e676 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:14.995749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9044a2d2-ad08-4a0c-b1d0-526d29f5e676 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:14.995985 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9044a2d2-ad08-4a0c-b1d0-526d29f5e676 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:15.030373 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9044a2d2-ad08-4a0c-b1d0-526d29f5e676 None None] MFA Rules not processed for user `7d360f8d42634cb88e17acbb3aaab276`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:15.031639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9044a2d2-ad08-4a0c-b1d0-526d29f5e676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:15.107267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1241/2495] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:14 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:15.111343 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca5c5ad-590d-4b9f-842f-422a1f14fb3c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:15.175515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:15.191159 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca5c5ad-590d-4b9f-842f-422a1f14fb3c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:15.197142 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb58d1c-1dd1-4af8-82ea-30f062987537 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:15.198201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb58d1c-1dd1-4af8-82ea-30f062987537 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:15.198709 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb58d1c-1dd1-4af8-82ea-30f062987537 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:15.199059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb58d1c-1dd1-4af8-82ea-30f062987537 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:15.199810 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb58d1c-1dd1-4af8-82ea-30f062987537 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:15.201769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb58d1c-1dd1-4af8-82ea-30f062987537 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:15.239270 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1255/2496] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:25:15 2022] POST /identity/v3/projects => generated 405 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:15.240424 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9044a2d2-ad08-4a0c-b1d0-526d29f5e676 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:15.256547 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9044a2d2-ad08-4a0c-b1d0-526d29f5e676 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:15.305254 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb58d1c-1dd1-4af8-82ea-30f062987537 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:15.308078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-1bb58d1c-1dd1-4af8-82ea-30f062987537 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:15.343022 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c23097c-0963-4471-a35e-27d9d1baa8f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:15.344081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c23097c-0963-4471-a35e-27d9d1baa8f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:15.344081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c23097c-0963-4471-a35e-27d9d1baa8f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:15.367426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c23097c-0963-4471-a35e-27d9d1baa8f5 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:15.369424 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c23097c-0963-4471-a35e-27d9d1baa8f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:15.374731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c23097c-0963-4471-a35e-27d9d1baa8f5 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:15.389820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb58d1c-1dd1-4af8-82ea-30f062987537 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:15.395015 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1b4a62-6e7a-4582-9952-4ec444359465 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:15.396203 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1b4a62-6e7a-4582-9952-4ec444359465 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:15.396203 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1b4a62-6e7a-4582-9952-4ec444359465 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:15.396418 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1b4a62-6e7a-4582-9952-4ec444359465 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:15.400356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6d1b4a62-6e7a-4582-9952-4ec444359465 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:15.423770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1242/2497] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:25:15 2022] POST /identity/v3/users => generated 309 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:25:15.427842 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c23097c-0963-4471-a35e-27d9d1baa8f5 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:15.448098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c23097c-0963-4471-a35e-27d9d1baa8f5 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:15.456416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371a0a88-34e2-4752-b9f6-d0dbe7aa6e8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:15.457536 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371a0a88-34e2-4752-b9f6-d0dbe7aa6e8e admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:15.458122 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371a0a88-34e2-4752-b9f6-d0dbe7aa6e8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:15.458822 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371a0a88-34e2-4752-b9f6-d0dbe7aa6e8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:15.460642 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371a0a88-34e2-4752-b9f6-d0dbe7aa6e8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:15.465323 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371a0a88-34e2-4752-b9f6-d0dbe7aa6e8e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:15.467813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:15.494817 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1243/2498] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:15 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:15.499369 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371a0a88-34e2-4752-b9f6-d0dbe7aa6e8e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:15.502753 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1b4a62-6e7a-4582-9952-4ec444359465 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:15.504342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1b4a62-6e7a-4582-9952-4ec444359465 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:15.525650 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371a0a88-34e2-4752-b9f6-d0dbe7aa6e8e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:15.529250 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1256/2499] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:15 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:15.533095 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badbade3-2246-4fa8-818c-fe9ecc95da85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:15.533983 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badbade3-2246-4fa8-818c-fe9ecc95da85 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:15.534280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badbade3-2246-4fa8-818c-fe9ecc95da85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:15.534543 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badbade3-2246-4fa8-818c-fe9ecc95da85 admin admin] PATH_INFO: `/v3/projects/6c9b2b68977b4eafb4c2d773cf90e123/users/023046d36aaa49f6a04054b7364f3816/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:15.554373 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badbade3-2246-4fa8-818c-fe9ecc95da85 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c9b2b68977b4eafb4c2d773cf90e123, user_id=023046d36aaa49f6a04054b7364f3816, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:15.559306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badbade3-2246-4fa8-818c-fe9ecc95da85 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:15.582923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1244/2500] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:15 2022] PUT /identity/v3/projects/6c9b2b68977b4eafb4c2d773cf90e123/users/023046d36aaa49f6a04054b7364f3816/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:15.588248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1b4a62-6e7a-4582-9952-4ec444359465 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:15.603562 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1b4a62-6e7a-4582-9952-4ec444359465 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:15.669937 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef76f4c-1894-4516-8256-c12a4ef6ce8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:15.670731 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef76f4c-1894-4516-8256-c12a4ef6ce8f admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:15.671151 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef76f4c-1894-4516-8256-c12a4ef6ce8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:15.671522 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef76f4c-1894-4516-8256-c12a4ef6ce8f admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:15.672255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef76f4c-1894-4516-8256-c12a4ef6ce8f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:15.673947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef76f4c-1894-4516-8256-c12a4ef6ce8f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:15.689543 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1257/2501] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:15 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:15.690183 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badbade3-2246-4fa8-818c-fe9ecc95da85 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:15.715087 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badbade3-2246-4fa8-818c-fe9ecc95da85 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:15.730157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa853c32-a580-4953-bece-5dd14adc3ebc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:15.735741 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa853c32-a580-4953-bece-5dd14adc3ebc admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:15.736056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa853c32-a580-4953-bece-5dd14adc3ebc admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:15.736363 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa853c32-a580-4953-bece-5dd14adc3ebc admin admin] PATH_INFO: `/v3/projects/6c9b2b68977b4eafb4c2d773cf90e123/users/023046d36aaa49f6a04054b7364f3816/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:15.738582 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa853c32-a580-4953-bece-5dd14adc3ebc admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c9b2b68977b4eafb4c2d773cf90e123, user_id=023046d36aaa49f6a04054b7364f3816, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:15.741238 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa853c32-a580-4953-bece-5dd14adc3ebc admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:15.775009 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1245/2502] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:15 2022] PUT /identity/v3/projects/6c9b2b68977b4eafb4c2d773cf90e123/users/023046d36aaa49f6a04054b7364f3816/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:25:15.778922 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9febbe20-3524-47fd-8e51-7ab169a959b2 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:15.779505 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9febbe20-3524-47fd-8e51-7ab169a959b2 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:15.780000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9febbe20-3524-47fd-8e51-7ab169a959b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:15.832409 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-9febbe20-3524-47fd-8e51-7ab169a959b2 None None] MFA Rules not processed for user `023046d36aaa49f6a04054b7364f3816`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:15.833937 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9febbe20-3524-47fd-8e51-7ab169a959b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:15.856892 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:15.957306 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1258/2503] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:15 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:15.958227 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa853c32-a580-4953-bece-5dd14adc3ebc admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:15.960568 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fa853c32-a580-4953-bece-5dd14adc3ebc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:16.029304 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa853c32-a580-4953-bece-5dd14adc3ebc admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:16.033826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb99dd6-57aa-477d-8954-f86d5732bbd0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:16.034681 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb99dd6-57aa-477d-8954-f86d5732bbd0 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:16.035069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb99dd6-57aa-477d-8954-f86d5732bbd0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:16.035409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb99dd6-57aa-477d-8954-f86d5732bbd0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:16.046571 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb99dd6-57aa-477d-8954-f86d5732bbd0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:16.048354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb99dd6-57aa-477d-8954-f86d5732bbd0 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:16.072165 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1246/2504] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:15 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:16.221273 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9febbe20-3524-47fd-8e51-7ab169a959b2 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:16.302954 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9febbe20-3524-47fd-8e51-7ab169a959b2 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:16.309093 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94714170-a536-4f1d-a577-62e1746d928f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:16.310384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94714170-a536-4f1d-a577-62e1746d928f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:16.310820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94714170-a536-4f1d-a577-62e1746d928f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:16.311613 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94714170-a536-4f1d-a577-62e1746d928f admin admin] PATH_INFO: `/v3/users/7d360f8d42634cb88e17acbb3aaab276` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:16.313271 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94714170-a536-4f1d-a577-62e1746d928f admin admin] RBAC: Authorizing `identity:delete_user(user_id=7d360f8d42634cb88e17acbb3aaab276)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:16.315244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94714170-a536-4f1d-a577-62e1746d928f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:16.377808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-94714170-a536-4f1d-a577-62e1746d928f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7d360f8d42634cb88e17acbb3aaab276'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:16.378529 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-94714170-a536-4f1d-a577-62e1746d928f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7d360f8d42634cb88e17acbb3aaab276'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:16.391040 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-94714170-a536-4f1d-a577-62e1746d928f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7d360f8d42634cb88e17acbb3aaab276'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:16.391682 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-94714170-a536-4f1d-a577-62e1746d928f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7d360f8d42634cb88e17acbb3aaab276'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:16.418213 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-94714170-a536-4f1d-a577-62e1746d928f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7d360f8d42634cb88e17acbb3aaab276'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:16.436438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1259/2505] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:25:16 2022] DELETE /identity/v3/users/7d360f8d42634cb88e17acbb3aaab276 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:16.439601 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d513168-dfa3-45f4-80d5-2c99451d35fa None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:16.439901 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d513168-dfa3-45f4-80d5-2c99451d35fa None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:16.440818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d513168-dfa3-45f4-80d5-2c99451d35fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:16.462735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-5d513168-dfa3-45f4-80d5-2c99451d35fa None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:16.464116 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5d513168-dfa3-45f4-80d5-2c99451d35fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:16.551941 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1247/2506] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:25:16.562580 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94714170-a536-4f1d-a577-62e1746d928f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:16.564574 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-94714170-a536-4f1d-a577-62e1746d928f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:16.675429 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:16.705639 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94714170-a536-4f1d-a577-62e1746d928f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:16.709338 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a563fa-fe3d-436a-95aa-bd051524c179 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:16.709949 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a563fa-fe3d-436a-95aa-bd051524c179 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:16.710222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a563fa-fe3d-436a-95aa-bd051524c179 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:16.710631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a563fa-fe3d-436a-95aa-bd051524c179 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:16.721127 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a563fa-fe3d-436a-95aa-bd051524c179 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:16.722718 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a563fa-fe3d-436a-95aa-bd051524c179 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:16.735912 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1260/2507] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:17.293916 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d513168-dfa3-45f4-80d5-2c99451d35fa None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:17.295693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5d513168-dfa3-45f4-80d5-2c99451d35fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:17.319155 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d513168-dfa3-45f4-80d5-2c99451d35fa None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:17.338426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e67403d-9e55-44cb-ad39-e042ff4530b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:17.340110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e67403d-9e55-44cb-ad39-e042ff4530b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:17.340392 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e67403d-9e55-44cb-ad39-e042ff4530b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:17.340667 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e67403d-9e55-44cb-ad39-e042ff4530b1 admin admin] PATH_INFO: `/v3/projects/bc5cf717227340829fa38e3159cf8c35` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:17.343218 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e67403d-9e55-44cb-ad39-e042ff4530b1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bc5cf717227340829fa38e3159cf8c35)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:17.345850 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e67403d-9e55-44cb-ad39-e042ff4530b1 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:17.434538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4e67403d-9e55-44cb-ad39-e042ff4530b1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bc5cf717227340829fa38e3159cf8c35', 'request_id': 'req-4e67403d-9e55-44cb-ad39-e042ff4530b1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:17.439807 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:17.441890 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4e67403d-9e55-44cb-ad39-e042ff4530b1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bc5cf717227340829fa38e3159cf8c35', 'request_id': 'req-4e67403d-9e55-44cb-ad39-e042ff4530b1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:17.451269 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4e67403d-9e55-44cb-ad39-e042ff4530b1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bc5cf717227340829fa38e3159cf8c35', 'request_id': 'req-4e67403d-9e55-44cb-ad39-e042ff4530b1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:17.474703 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4e67403d-9e55-44cb-ad39-e042ff4530b1 admin admin] The token cache is being invalidate because project bc5cf717227340829fa38e3159cf8c35 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:25:17.475111 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4e67403d-9e55-44cb-ad39-e042ff4530b1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:17.475872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4e67403d-9e55-44cb-ad39-e042ff4530b1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:17.480549 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1248/2508] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:25:17 2022] DELETE /identity/v3/projects/bc5cf717227340829fa38e3159cf8c35 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:17.484966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a563fa-fe3d-436a-95aa-bd051524c179 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:17.496917 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a563fa-fe3d-436a-95aa-bd051524c179 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:17.574823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a41f07-a489-4606-8956-c4bfd907c999 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:17.585111 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a41f07-a489-4606-8956-c4bfd907c999 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:17.585449 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a41f07-a489-4606-8956-c4bfd907c999 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:17.588091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a41f07-a489-4606-8956-c4bfd907c999 admin admin] PATH_INFO: `/v3/users/023046d36aaa49f6a04054b7364f3816` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:17.588091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a41f07-a489-4606-8956-c4bfd907c999 admin admin] RBAC: Authorizing `identity:delete_user(user_id=023046d36aaa49f6a04054b7364f3816)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:17.590633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a41f07-a489-4606-8956-c4bfd907c999 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:17.695999 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f4a41f07-a489-4606-8956-c4bfd907c999 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '023046d36aaa49f6a04054b7364f3816'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:17.697010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f4a41f07-a489-4606-8956-c4bfd907c999 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '023046d36aaa49f6a04054b7364f3816'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:17.706441 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f4a41f07-a489-4606-8956-c4bfd907c999 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '023046d36aaa49f6a04054b7364f3816'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:17.708073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f4a41f07-a489-4606-8956-c4bfd907c999 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '023046d36aaa49f6a04054b7364f3816'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:17.725660 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f4a41f07-a489-4606-8956-c4bfd907c999 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '023046d36aaa49f6a04054b7364f3816'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:17.743591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1261/2509] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:25:17 2022] DELETE /identity/v3/users/023046d36aaa49f6a04054b7364f3816 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:18.123070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e67403d-9e55-44cb-ad39-e042ff4530b1 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:18.125317 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4e67403d-9e55-44cb-ad39-e042ff4530b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:18.157432 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e67403d-9e55-44cb-ad39-e042ff4530b1 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:18.249057 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e677102-7750-4ad0-8811-87ddfd4ab1cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:18.249830 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e677102-7750-4ad0-8811-87ddfd4ab1cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:18.250178 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e677102-7750-4ad0-8811-87ddfd4ab1cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:18.250541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e677102-7750-4ad0-8811-87ddfd4ab1cd admin admin] PATH_INFO: `/v3/projects/6c9b2b68977b4eafb4c2d773cf90e123` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:18.252147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e677102-7750-4ad0-8811-87ddfd4ab1cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=6c9b2b68977b4eafb4c2d773cf90e123)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:18.254356 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e677102-7750-4ad0-8811-87ddfd4ab1cd admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:18.312707 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4e677102-7750-4ad0-8811-87ddfd4ab1cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6c9b2b68977b4eafb4c2d773cf90e123', 'request_id': 'req-4e677102-7750-4ad0-8811-87ddfd4ab1cd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:18.328182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4e677102-7750-4ad0-8811-87ddfd4ab1cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6c9b2b68977b4eafb4c2d773cf90e123', 'request_id': 'req-4e677102-7750-4ad0-8811-87ddfd4ab1cd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:18.348335 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4e677102-7750-4ad0-8811-87ddfd4ab1cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6c9b2b68977b4eafb4c2d773cf90e123', 'request_id': 'req-4e677102-7750-4ad0-8811-87ddfd4ab1cd'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:18.366894 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4e677102-7750-4ad0-8811-87ddfd4ab1cd admin admin] The token cache is being invalidate because project 6c9b2b68977b4eafb4c2d773cf90e123 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:25:18.367328 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4e677102-7750-4ad0-8811-87ddfd4ab1cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:18.368090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4e677102-7750-4ad0-8811-87ddfd4ab1cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:18.370106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1249/2510] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:25:18 2022] DELETE /identity/v3/projects/6c9b2b68977b4eafb4c2d773cf90e123 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:18.375631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035d250e-5d52-416a-a717-9ed6a8069daa None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:18.376005 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035d250e-5d52-416a-a717-9ed6a8069daa None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:18.376504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035d250e-5d52-416a-a717-9ed6a8069daa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:18.403230 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-035d250e-5d52-416a-a717-9ed6a8069daa None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:18.404473 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-035d250e-5d52-416a-a717-9ed6a8069daa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:18.464406 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:18.483796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1262/2511] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:18.493179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfd0b93-c195-4814-ae4c-3253bf5ded61 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:18.493460 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfd0b93-c195-4814-ae4c-3253bf5ded61 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:18.493729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfd0b93-c195-4814-ae4c-3253bf5ded61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:18.519935 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-0dfd0b93-c195-4814-ae4c-3253bf5ded61 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:18.521269 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0dfd0b93-c195-4814-ae4c-3253bf5ded61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:18.535040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1250/2512] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:18.536998 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035d250e-5d52-416a-a717-9ed6a8069daa None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:18.564135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035d250e-5d52-416a-a717-9ed6a8069daa None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:18.579889 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75514e3a-c293-46ed-82b1-7866876e4dd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:18.580537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75514e3a-c293-46ed-82b1-7866876e4dd9 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:18.580824 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75514e3a-c293-46ed-82b1-7866876e4dd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:18.581104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75514e3a-c293-46ed-82b1-7866876e4dd9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:18.581777 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75514e3a-c293-46ed-82b1-7866876e4dd9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:25:18.582124 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75514e3a-c293-46ed-82b1-7866876e4dd9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:18.583735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75514e3a-c293-46ed-82b1-7866876e4dd9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:18.610341 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1263/2513] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:25:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:25:18.615201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dfd0b93-c195-4814-ae4c-3253bf5ded61 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:18.625934 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dfd0b93-c195-4814-ae4c-3253bf5ded61 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:18.632008 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e102bf4-fb60-4cb3-a387-37f72a6435fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:18.633413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e102bf4-fb60-4cb3-a387-37f72a6435fd admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:18.633686 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e102bf4-fb60-4cb3-a387-37f72a6435fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:18.633946 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e102bf4-fb60-4cb3-a387-37f72a6435fd admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:18.634581 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e102bf4-fb60-4cb3-a387-37f72a6435fd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:18.637133 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e102bf4-fb60-4cb3-a387-37f72a6435fd admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:18.671720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1251/2514] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:25:18 2022] POST /identity/v3/projects => generated 365 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:18.675547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75514e3a-c293-46ed-82b1-7866876e4dd9 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:18.687414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75514e3a-c293-46ed-82b1-7866876e4dd9 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:18.776327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdee8dd-9362-4afe-802f-79b00ebcc57c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:18.777014 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdee8dd-9362-4afe-802f-79b00ebcc57c admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:18.777301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdee8dd-9362-4afe-802f-79b00ebcc57c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:18.777584 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdee8dd-9362-4afe-802f-79b00ebcc57c admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:18.778212 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdee8dd-9362-4afe-802f-79b00ebcc57c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:18.780160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdee8dd-9362-4afe-802f-79b00ebcc57c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:18.827049 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1264/2515] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:25:18 2022] POST /identity/v3/users => generated 283 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:18.828275 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e102bf4-fb60-4cb3-a387-37f72a6435fd admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:18.840874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e102bf4-fb60-4cb3-a387-37f72a6435fd admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:18.845924 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d448245-d37c-4331-be2e-407e74fe0f2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:18.846681 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d448245-d37c-4331-be2e-407e74fe0f2c admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:18.847002 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d448245-d37c-4331-be2e-407e74fe0f2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:18.847342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d448245-d37c-4331-be2e-407e74fe0f2c admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:18.848013 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d448245-d37c-4331-be2e-407e74fe0f2c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:18.849620 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d448245-d37c-4331-be2e-407e74fe0f2c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:18.863882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1252/2516] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:18 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:18.865279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdee8dd-9362-4afe-802f-79b00ebcc57c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:18.880798 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdee8dd-9362-4afe-802f-79b00ebcc57c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:18.887281 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453b347e-2ff4-4986-af1a-080bd618c484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:18.887921 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453b347e-2ff4-4986-af1a-080bd618c484 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:18.888206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453b347e-2ff4-4986-af1a-080bd618c484 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:18.888484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453b347e-2ff4-4986-af1a-080bd618c484 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:18.889126 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453b347e-2ff4-4986-af1a-080bd618c484 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:18.890731 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453b347e-2ff4-4986-af1a-080bd618c484 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:18.909374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1265/2517] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:18 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:18.910892 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d448245-d37c-4331-be2e-407e74fe0f2c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:18.932535 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d448245-d37c-4331-be2e-407e74fe0f2c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:18.943663 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f025b92-d9d1-47d0-932c-c528ce049dc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:18.944416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f025b92-d9d1-47d0-932c-c528ce049dc0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:18.944769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f025b92-d9d1-47d0-932c-c528ce049dc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:18.946395 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f025b92-d9d1-47d0-932c-c528ce049dc0 admin admin] PATH_INFO: `/v3/projects/624485a1a4674b64a490f9893a90cd33/users/191a557f9ad34d569232a666fc6cf5e4/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:18.964301 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f025b92-d9d1-47d0-932c-c528ce049dc0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=624485a1a4674b64a490f9893a90cd33, user_id=191a557f9ad34d569232a666fc6cf5e4, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:18.967854 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f025b92-d9d1-47d0-932c-c528ce049dc0 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:18.987186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1253/2518] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:18 2022] PUT /identity/v3/projects/624485a1a4674b64a490f9893a90cd33/users/191a557f9ad34d569232a666fc6cf5e4/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:19.000209 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede03ea1-b9b6-48c3-b930-f5ce456deff7 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:19.000496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede03ea1-b9b6-48c3-b930-f5ce456deff7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:19.002230 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede03ea1-b9b6-48c3-b930-f5ce456deff7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:19.050484 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ede03ea1-b9b6-48c3-b930-f5ce456deff7 None None] MFA Rules not processed for user `191a557f9ad34d569232a666fc6cf5e4`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:19.052797 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ede03ea1-b9b6-48c3-b930-f5ce456deff7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:19.151050 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1266/2519] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:18 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:19.152885 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f025b92-d9d1-47d0-932c-c528ce049dc0 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:19.171286 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:19.246629 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f025b92-d9d1-47d0-932c-c528ce049dc0 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:19.255405 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb9ecc4-e9d2-4561-a2ba-f090bac67110 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:19.256416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb9ecc4-e9d2-4561-a2ba-f090bac67110 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:19.256880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb9ecc4-e9d2-4561-a2ba-f090bac67110 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:19.257378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb9ecc4-e9d2-4561-a2ba-f090bac67110 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:19.258247 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb9ecc4-e9d2-4561-a2ba-f090bac67110 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:19.260425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb9ecc4-e9d2-4561-a2ba-f090bac67110 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:19.301139 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1254/2520] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:25:19 2022] POST /identity/v3/projects => generated 365 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:19.306494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede03ea1-b9b6-48c3-b930-f5ce456deff7 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:19.320013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede03ea1-b9b6-48c3-b930-f5ce456deff7 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:19.389513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:19.411319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2018fa-341e-4f7b-9be3-16fabf91e0cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:19.411998 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2018fa-341e-4f7b-9be3-16fabf91e0cd admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:19.412348 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2018fa-341e-4f7b-9be3-16fabf91e0cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:19.412712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2018fa-341e-4f7b-9be3-16fabf91e0cd admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:19.413422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2018fa-341e-4f7b-9be3-16fabf91e0cd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:19.428494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2018fa-341e-4f7b-9be3-16fabf91e0cd admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:19.477365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1267/2521] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:25:19 2022] POST /identity/v3/users => generated 289 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:25:19.478295 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb9ecc4-e9d2-4561-a2ba-f090bac67110 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:19.494671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb9ecc4-e9d2-4561-a2ba-f090bac67110 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:19.501380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755a60d5-2782-4c0d-9bcd-cbbaf7b09422 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:19.502155 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755a60d5-2782-4c0d-9bcd-cbbaf7b09422 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:19.502413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755a60d5-2782-4c0d-9bcd-cbbaf7b09422 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:19.502727 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755a60d5-2782-4c0d-9bcd-cbbaf7b09422 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:19.503399 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755a60d5-2782-4c0d-9bcd-cbbaf7b09422 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:19.505802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755a60d5-2782-4c0d-9bcd-cbbaf7b09422 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:19.521872 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2018fa-341e-4f7b-9be3-16fabf91e0cd admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:19.529780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1255/2522] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:19 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:19.533670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2018fa-341e-4f7b-9be3-16fabf91e0cd admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:19.539140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46b8e24-6722-407b-81ca-4a602ec7dec5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:19.539808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46b8e24-6722-407b-81ca-4a602ec7dec5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:19.540092 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46b8e24-6722-407b-81ca-4a602ec7dec5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:19.540372 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46b8e24-6722-407b-81ca-4a602ec7dec5 admin admin] PATH_INFO: `/v3/projects/4084d3ae34374a24be301694bd6c4f25/users/58cbc29e92aa4b7990f41cc94452f22d/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:19.559403 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46b8e24-6722-407b-81ca-4a602ec7dec5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4084d3ae34374a24be301694bd6c4f25, user_id=58cbc29e92aa4b7990f41cc94452f22d, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:19.562207 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46b8e24-6722-407b-81ca-4a602ec7dec5 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:19.581449 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1268/2523] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:19 2022] PUT /identity/v3/projects/4084d3ae34374a24be301694bd6c4f25/users/58cbc29e92aa4b7990f41cc94452f22d/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:19.586309 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755a60d5-2782-4c0d-9bcd-cbbaf7b09422 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:19.600936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755a60d5-2782-4c0d-9bcd-cbbaf7b09422 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:19.716296 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e6480f-873b-499b-b92c-3ee76e3fde53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:19.718276 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e6480f-873b-499b-b92c-3ee76e3fde53 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:19.719032 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e6480f-873b-499b-b92c-3ee76e3fde53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:19.719504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e6480f-873b-499b-b92c-3ee76e3fde53 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:19.720492 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e6480f-873b-499b-b92c-3ee76e3fde53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:19.722316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e6480f-873b-499b-b92c-3ee76e3fde53 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:19.735587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1256/2524] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:19 2022] GET /identity/v3/roles => generated 1420 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:19.739994 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46b8e24-6722-407b-81ca-4a602ec7dec5 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:19.752921 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46b8e24-6722-407b-81ca-4a602ec7dec5 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:19.758415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb6822b-6286-45df-bb2e-b0a155598e23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:19.759272 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb6822b-6286-45df-bb2e-b0a155598e23 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:19.759652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb6822b-6286-45df-bb2e-b0a155598e23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:19.760013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb6822b-6286-45df-bb2e-b0a155598e23 admin admin] PATH_INFO: `/v3/projects/4084d3ae34374a24be301694bd6c4f25/users/58cbc29e92aa4b7990f41cc94452f22d/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:19.774790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb6822b-6286-45df-bb2e-b0a155598e23 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4084d3ae34374a24be301694bd6c4f25, user_id=58cbc29e92aa4b7990f41cc94452f22d, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:19.787461 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb6822b-6286-45df-bb2e-b0a155598e23 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:19.814287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1269/2525] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:19 2022] PUT /identity/v3/projects/4084d3ae34374a24be301694bd6c4f25/users/58cbc29e92aa4b7990f41cc94452f22d/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:25:19.817102 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553ba2fc-fc8e-480a-903c-9bd3b5cee051 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:19.817662 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553ba2fc-fc8e-480a-903c-9bd3b5cee051 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:19.818112 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553ba2fc-fc8e-480a-903c-9bd3b5cee051 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:19.884104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-553ba2fc-fc8e-480a-903c-9bd3b5cee051 None None] MFA Rules not processed for user `58cbc29e92aa4b7990f41cc94452f22d`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:19.885621 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-553ba2fc-fc8e-480a-903c-9bd3b5cee051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:19.992666 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1257/2526] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:19 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:20.005566 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb6822b-6286-45df-bb2e-b0a155598e23 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:20.008820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9fb6822b-6286-45df-bb2e-b0a155598e23 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:20.107155 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb6822b-6286-45df-bb2e-b0a155598e23 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:20.113443 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8306c6e1-8111-4fe3-8122-b3c6b4495758 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:20.114592 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8306c6e1-8111-4fe3-8122-b3c6b4495758 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:20.115189 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8306c6e1-8111-4fe3-8122-b3c6b4495758 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:20.115649 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8306c6e1-8111-4fe3-8122-b3c6b4495758 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:20.134820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8306c6e1-8111-4fe3-8122-b3c6b4495758 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:20.137089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8306c6e1-8111-4fe3-8122-b3c6b4495758 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:20.156472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1270/2527] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:20 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:20.273906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553ba2fc-fc8e-480a-903c-9bd3b5cee051 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:20.285037 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553ba2fc-fc8e-480a-903c-9bd3b5cee051 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:20.289426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29f8a7f-f36f-4485-8507-1da3497c677c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:20.290072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29f8a7f-f36f-4485-8507-1da3497c677c service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:20.290365 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29f8a7f-f36f-4485-8507-1da3497c677c service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:20.295466 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29f8a7f-f36f-4485-8507-1da3497c677c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:20.295687 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:20.378521 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29f8a7f-f36f-4485-8507-1da3497c677c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:20.380038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29f8a7f-f36f-4485-8507-1da3497c677c service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:20.391223 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1258/2528] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:20 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:21.265023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8306c6e1-8111-4fe3-8122-b3c6b4495758 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:21.267054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8306c6e1-8111-4fe3-8122-b3c6b4495758 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:21.353188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8306c6e1-8111-4fe3-8122-b3c6b4495758 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:21.360119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087bcfcf-7b47-499e-8de4-1916982bb82e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:21.360119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087bcfcf-7b47-499e-8de4-1916982bb82e service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:21.360119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087bcfcf-7b47-499e-8de4-1916982bb82e service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:21.360119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087bcfcf-7b47-499e-8de4-1916982bb82e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:21.360806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-087bcfcf-7b47-499e-8de4-1916982bb82e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:21.430178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:21.435951 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087bcfcf-7b47-499e-8de4-1916982bb82e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:21.438096 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087bcfcf-7b47-499e-8de4-1916982bb82e service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:21.450364 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1271/2529] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:25:29.272215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29f8a7f-f36f-4485-8507-1da3497c677c service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:29.274888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c29f8a7f-f36f-4485-8507-1da3497c677c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:29.306653 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29f8a7f-f36f-4485-8507-1da3497c677c service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:29.318833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41ecced-9546-4f35-a87c-26d49fd8cb41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:29.320220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41ecced-9546-4f35-a87c-26d49fd8cb41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:29.320749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41ecced-9546-4f35-a87c-26d49fd8cb41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:29.321265 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41ecced-9546-4f35-a87c-26d49fd8cb41 admin admin] PATH_INFO: `/v3/users/85178dc2280c4a54970fc207d1746608` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:29.322752 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41ecced-9546-4f35-a87c-26d49fd8cb41 admin admin] RBAC: Authorizing `identity:delete_user(user_id=85178dc2280c4a54970fc207d1746608)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:29.324993 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41ecced-9546-4f35-a87c-26d49fd8cb41 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:29.436790 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a41ecced-9546-4f35-a87c-26d49fd8cb41 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '85178dc2280c4a54970fc207d1746608'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:29.437550 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a41ecced-9546-4f35-a87c-26d49fd8cb41 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '85178dc2280c4a54970fc207d1746608'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:29.448793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a41ecced-9546-4f35-a87c-26d49fd8cb41 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '85178dc2280c4a54970fc207d1746608'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:29.449302 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a41ecced-9546-4f35-a87c-26d49fd8cb41 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '85178dc2280c4a54970fc207d1746608'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:29.463322 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a41ecced-9546-4f35-a87c-26d49fd8cb41 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '85178dc2280c4a54970fc207d1746608'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:29.486944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1259/2530] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:25:29 2022] DELETE /identity/v3/users/85178dc2280c4a54970fc207d1746608 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:29.731959 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087bcfcf-7b47-499e-8de4-1916982bb82e service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:29.746121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087bcfcf-7b47-499e-8de4-1916982bb82e service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:29.877121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6288df2-9551-4512-8dcd-35f81a12553e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:29.878011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6288df2-9551-4512-8dcd-35f81a12553e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:29.878294 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6288df2-9551-4512-8dcd-35f81a12553e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:29.878806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6288df2-9551-4512-8dcd-35f81a12553e admin admin] PATH_INFO: `/v3/projects/de31bd722d4f419d9269b519af7f793a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:29.880200 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6288df2-9551-4512-8dcd-35f81a12553e admin admin] RBAC: Authorizing `identity:delete_project(project_id=de31bd722d4f419d9269b519af7f793a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:29.883065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6288df2-9551-4512-8dcd-35f81a12553e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:29.950263 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e6288df2-9551-4512-8dcd-35f81a12553e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'de31bd722d4f419d9269b519af7f793a', 'request_id': 'req-e6288df2-9551-4512-8dcd-35f81a12553e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:29.957720 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e6288df2-9551-4512-8dcd-35f81a12553e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'de31bd722d4f419d9269b519af7f793a', 'request_id': 'req-e6288df2-9551-4512-8dcd-35f81a12553e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:29.965574 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e6288df2-9551-4512-8dcd-35f81a12553e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'de31bd722d4f419d9269b519af7f793a', 'request_id': 'req-e6288df2-9551-4512-8dcd-35f81a12553e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:29.981654 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e6288df2-9551-4512-8dcd-35f81a12553e admin admin] The token cache is being invalidate because project de31bd722d4f419d9269b519af7f793a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:25:29.981958 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e6288df2-9551-4512-8dcd-35f81a12553e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:29.982642 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e6288df2-9551-4512-8dcd-35f81a12553e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:29.984408 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1272/2531] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:25:29 2022] DELETE /identity/v3/projects/de31bd722d4f419d9269b519af7f793a => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:29.988561 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41ecced-9546-4f35-a87c-26d49fd8cb41 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:29.990095 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a41ecced-9546-4f35-a87c-26d49fd8cb41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:30.011867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41ecced-9546-4f35-a87c-26d49fd8cb41 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:30.077572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9816d14e-612f-4ceb-9a8b-5914c199f4cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:30.078241 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9816d14e-612f-4ceb-9a8b-5914c199f4cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:30.078611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9816d14e-612f-4ceb-9a8b-5914c199f4cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:30.079003 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9816d14e-612f-4ceb-9a8b-5914c199f4cf admin admin] PATH_INFO: `/v3/users/ee8f5c0bb6c947dfbd35022ecfac08c4` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:30.080321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9816d14e-612f-4ceb-9a8b-5914c199f4cf admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee8f5c0bb6c947dfbd35022ecfac08c4)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:30.082323 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9816d14e-612f-4ceb-9a8b-5914c199f4cf admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:30.106672 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:30.136417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9816d14e-612f-4ceb-9a8b-5914c199f4cf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee8f5c0bb6c947dfbd35022ecfac08c4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:30.137393 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9816d14e-612f-4ceb-9a8b-5914c199f4cf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee8f5c0bb6c947dfbd35022ecfac08c4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:30.146074 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9816d14e-612f-4ceb-9a8b-5914c199f4cf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee8f5c0bb6c947dfbd35022ecfac08c4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:30.146844 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9816d14e-612f-4ceb-9a8b-5914c199f4cf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee8f5c0bb6c947dfbd35022ecfac08c4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:30.157809 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9816d14e-612f-4ceb-9a8b-5914c199f4cf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee8f5c0bb6c947dfbd35022ecfac08c4'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:30.178464 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1260/2532] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:25:29 2022] DELETE /identity/v3/users/ee8f5c0bb6c947dfbd35022ecfac08c4 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:30.248853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6288df2-9551-4512-8dcd-35f81a12553e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:30.250364 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e6288df2-9551-4512-8dcd-35f81a12553e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:30.347030 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6288df2-9551-4512-8dcd-35f81a12553e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:30.380565 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:30.393119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9816d14e-612f-4ceb-9a8b-5914c199f4cf admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:30.395050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9816d14e-612f-4ceb-9a8b-5914c199f4cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:30.463494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58465af-4c39-40c3-bf6c-827a15db434e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:30.464135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58465af-4c39-40c3-bf6c-827a15db434e service nova] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:30.464415 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58465af-4c39-40c3-bf6c-827a15db434e service nova] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:30.464688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58465af-4c39-40c3-bf6c-827a15db434e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:30.466471 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d58465af-4c39-40c3-bf6c-827a15db434e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:30.498084 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9816d14e-612f-4ceb-9a8b-5914c199f4cf admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:30.508889 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc388f83-42d5-4a45-81ab-12483511a2a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:30.510023 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc388f83-42d5-4a45-81ab-12483511a2a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:30.510717 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc388f83-42d5-4a45-81ab-12483511a2a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:30.511202 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc388f83-42d5-4a45-81ab-12483511a2a8 admin admin] PATH_INFO: `/v3/projects/bf6e7bbb274340ecbc5025be563f7524` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:30.512647 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc388f83-42d5-4a45-81ab-12483511a2a8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bf6e7bbb274340ecbc5025be563f7524)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:30.515130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc388f83-42d5-4a45-81ab-12483511a2a8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:30.546914 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58465af-4c39-40c3-bf6c-827a15db434e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:30.548515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58465af-4c39-40c3-bf6c-827a15db434e service nova] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:30.562632 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1273/2533] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:30 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:30.582586 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cc388f83-42d5-4a45-81ab-12483511a2a8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bf6e7bbb274340ecbc5025be563f7524', 'request_id': 'req-cc388f83-42d5-4a45-81ab-12483511a2a8'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:30.590071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cc388f83-42d5-4a45-81ab-12483511a2a8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bf6e7bbb274340ecbc5025be563f7524', 'request_id': 'req-cc388f83-42d5-4a45-81ab-12483511a2a8'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:30.598167 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cc388f83-42d5-4a45-81ab-12483511a2a8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bf6e7bbb274340ecbc5025be563f7524', 'request_id': 'req-cc388f83-42d5-4a45-81ab-12483511a2a8'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:30.621375 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cc388f83-42d5-4a45-81ab-12483511a2a8 admin admin] The token cache is being invalidate because project bf6e7bbb274340ecbc5025be563f7524 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:25:30.622017 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cc388f83-42d5-4a45-81ab-12483511a2a8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:30.627247 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-cc388f83-42d5-4a45-81ab-12483511a2a8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:30.628969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1261/2534] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:25:30 2022] DELETE /identity/v3/projects/bf6e7bbb274340ecbc5025be563f7524 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:32.495723 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58465af-4c39-40c3-bf6c-827a15db434e service nova] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:32.497220 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d58465af-4c39-40c3-bf6c-827a15db434e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:32.513934 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58465af-4c39-40c3-bf6c-827a15db434e service nova] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:32.591243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eaba8b1-c6b3-4a8b-b699-c9f066ac28df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:32.591982 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eaba8b1-c6b3-4a8b-b699-c9f066ac28df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:32.592289 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eaba8b1-c6b3-4a8b-b699-c9f066ac28df admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:32.592670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eaba8b1-c6b3-4a8b-b699-c9f066ac28df admin admin] PATH_INFO: `/v3/users/191a557f9ad34d569232a666fc6cf5e4` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:32.594094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eaba8b1-c6b3-4a8b-b699-c9f066ac28df admin admin] RBAC: Authorizing `identity:delete_user(user_id=191a557f9ad34d569232a666fc6cf5e4)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:32.597429 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eaba8b1-c6b3-4a8b-b699-c9f066ac28df admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:32.666996 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0eaba8b1-c6b3-4a8b-b699-c9f066ac28df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '191a557f9ad34d569232a666fc6cf5e4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:32.667673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0eaba8b1-c6b3-4a8b-b699-c9f066ac28df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '191a557f9ad34d569232a666fc6cf5e4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:32.677874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0eaba8b1-c6b3-4a8b-b699-c9f066ac28df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '191a557f9ad34d569232a666fc6cf5e4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:32.678772 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0eaba8b1-c6b3-4a8b-b699-c9f066ac28df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '191a557f9ad34d569232a666fc6cf5e4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:32.697306 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:32.708700 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0eaba8b1-c6b3-4a8b-b699-c9f066ac28df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '191a557f9ad34d569232a666fc6cf5e4'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:32.724199 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1274/2535] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:25:32 2022] DELETE /identity/v3/users/191a557f9ad34d569232a666fc6cf5e4 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:32.728715 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7e6ac8-c60e-4d0d-b648-a1c9c0704991 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:32.729188 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7e6ac8-c60e-4d0d-b648-a1c9c0704991 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:32.729411 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7e6ac8-c60e-4d0d-b648-a1c9c0704991 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:32.751814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9e7e6ac8-c60e-4d0d-b648-a1c9c0704991 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:32.753309 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9e7e6ac8-c60e-4d0d-b648-a1c9c0704991 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:32.780338 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:DELETE=15,keystone:SELECT=83,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:32.827515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1262/2536] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:32.837723 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eaba8b1-c6b3-4a8b-b699-c9f066ac28df admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:32.839274 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0eaba8b1-c6b3-4a8b-b699-c9f066ac28df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:32.896798 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eaba8b1-c6b3-4a8b-b699-c9f066ac28df admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:32.901397 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15270337-6207-4b6e-9bf2-9691b04ece23 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:32.902214 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15270337-6207-4b6e-9bf2-9691b04ece23 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:32.902589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15270337-6207-4b6e-9bf2-9691b04ece23 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:32.902954 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15270337-6207-4b6e-9bf2-9691b04ece23 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:32.912569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15270337-6207-4b6e-9bf2-9691b04ece23 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:32.914051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15270337-6207-4b6e-9bf2-9691b04ece23 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:32.923412 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1275/2537] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:33.221173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7e6ac8-c60e-4d0d-b648-a1c9c0704991 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:33.222721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9e7e6ac8-c60e-4d0d-b648-a1c9c0704991 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:33.236737 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7e6ac8-c60e-4d0d-b648-a1c9c0704991 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:33.245711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1372d3-892d-458f-8de9-9ee8574de741 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:33.246358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1372d3-892d-458f-8de9-9ee8574de741 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:33.246645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1372d3-892d-458f-8de9-9ee8574de741 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:33.246952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1372d3-892d-458f-8de9-9ee8574de741 admin admin] PATH_INFO: `/v3/projects/624485a1a4674b64a490f9893a90cd33` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:33.248632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1372d3-892d-458f-8de9-9ee8574de741 admin admin] RBAC: Authorizing `identity:delete_project(project_id=624485a1a4674b64a490f9893a90cd33)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:33.251477 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1372d3-892d-458f-8de9-9ee8574de741 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:33.308631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d120c85-c819-45c3-8f03-0329b08c785f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:33.308971 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d120c85-c819-45c3-8f03-0329b08c785f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:33.309179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d120c85-c819-45c3-8f03-0329b08c785f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:33.313415 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6b1372d3-892d-458f-8de9-9ee8574de741 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '624485a1a4674b64a490f9893a90cd33', 'request_id': 'req-6b1372d3-892d-458f-8de9-9ee8574de741'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:33.321383 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6b1372d3-892d-458f-8de9-9ee8574de741 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '624485a1a4674b64a490f9893a90cd33', 'request_id': 'req-6b1372d3-892d-458f-8de9-9ee8574de741'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:33.328759 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6b1372d3-892d-458f-8de9-9ee8574de741 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '624485a1a4674b64a490f9893a90cd33', 'request_id': 'req-6b1372d3-892d-458f-8de9-9ee8574de741'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:33.330046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-5d120c85-c819-45c3-8f03-0329b08c785f None None] MFA Rules not processed for user `e28a0ba45389417ba8ae92376dba87ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:33.331283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5d120c85-c819-45c3-8f03-0329b08c785f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:33.343329 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6b1372d3-892d-458f-8de9-9ee8574de741 admin admin] The token cache is being invalidate because project 624485a1a4674b64a490f9893a90cd33 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:25:33.343925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6b1372d3-892d-458f-8de9-9ee8574de741 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:33.344530 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6b1372d3-892d-458f-8de9-9ee8574de741 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:33.347464 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1263/2538] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:25:33 2022] DELETE /identity/v3/projects/624485a1a4674b64a490f9893a90cd33 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:33.352293 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1372d3-892d-458f-8de9-9ee8574de741 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:33.354335 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6b1372d3-892d-458f-8de9-9ee8574de741 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:33.399084 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1276/2539] 10.222.0.15 () {58 vars in 967 bytes} [Tue Sep 13 01:25:33 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:25:33.415721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d120c85-c819-45c3-8f03-0329b08c785f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:33.417242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5d120c85-c819-45c3-8f03-0329b08c785f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:33.450774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1372d3-892d-458f-8de9-9ee8574de741 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:33.459564 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d48fe8d-6577-466c-bec0-8efc5296ecda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:33.460490 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d48fe8d-6577-466c-bec0-8efc5296ecda admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:33.460778 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d48fe8d-6577-466c-bec0-8efc5296ecda admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:33.461129 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d48fe8d-6577-466c-bec0-8efc5296ecda admin admin] PATH_INFO: `/v3/users/58cbc29e92aa4b7990f41cc94452f22d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:33.463180 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d48fe8d-6577-466c-bec0-8efc5296ecda admin admin] RBAC: Authorizing `identity:delete_user(user_id=58cbc29e92aa4b7990f41cc94452f22d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:33.465171 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d48fe8d-6577-466c-bec0-8efc5296ecda admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:33.525599 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-8d48fe8d-6577-466c-bec0-8efc5296ecda admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '58cbc29e92aa4b7990f41cc94452f22d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:33.526397 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-8d48fe8d-6577-466c-bec0-8efc5296ecda admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '58cbc29e92aa4b7990f41cc94452f22d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:33.530153 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:33.533940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d120c85-c819-45c3-8f03-0329b08c785f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:33.535936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-8d48fe8d-6577-466c-bec0-8efc5296ecda admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '58cbc29e92aa4b7990f41cc94452f22d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:33.539273 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-8d48fe8d-6577-466c-bec0-8efc5296ecda admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '58cbc29e92aa4b7990f41cc94452f22d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:33.540233 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bddcd748-39b3-43c3-b258-7f740526decd service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d7d8ed4ede4ddca192fce0a497689d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:33.540859 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddcd748-39b3-43c3-b258-7f740526decd service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:33.541189 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddcd748-39b3-43c3-b258-7f740526decd service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:33.541670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddcd748-39b3-43c3-b258-7f740526decd service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:33.544295 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-bddcd748-39b3-43c3-b258-7f740526decd service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:33.551052 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-8d48fe8d-6577-466c-bec0-8efc5296ecda admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '58cbc29e92aa4b7990f41cc94452f22d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:33.556661 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bddcd748-39b3-43c3-b258-7f740526decd service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:33.558245 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bddcd748-39b3-43c3-b258-7f740526decd service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:33.567492 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1277/2540] 10.222.0.15 () {60 vars in 1384 bytes} [Tue Sep 13 01:25:33 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 01:25:33.579455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1264/2541] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:25:33 2022] DELETE /identity/v3/users/58cbc29e92aa4b7990f41cc94452f22d => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:33.924613 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bddcd748-39b3-43c3-b258-7f740526decd service swift] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:33.926049 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-bddcd748-39b3-43c3-b258-7f740526decd service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:33.970290 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bddcd748-39b3-43c3-b258-7f740526decd service swift] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:33.983744 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:34.054417 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500eda30-df5e-4944-b65b-7d30886e537d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:34.055090 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500eda30-df5e-4944-b65b-7d30886e537d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:34.055655 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500eda30-df5e-4944-b65b-7d30886e537d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:34.055925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500eda30-df5e-4944-b65b-7d30886e537d admin admin] PATH_INFO: `/v3/projects/4084d3ae34374a24be301694bd6c4f25` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:34.057062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500eda30-df5e-4944-b65b-7d30886e537d admin admin] RBAC: Authorizing `identity:delete_project(project_id=4084d3ae34374a24be301694bd6c4f25)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:34.059164 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500eda30-df5e-4944-b65b-7d30886e537d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:34.114145 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-500eda30-df5e-4944-b65b-7d30886e537d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4084d3ae34374a24be301694bd6c4f25', 'request_id': 'req-500eda30-df5e-4944-b65b-7d30886e537d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:34.121303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-500eda30-df5e-4944-b65b-7d30886e537d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4084d3ae34374a24be301694bd6c4f25', 'request_id': 'req-500eda30-df5e-4944-b65b-7d30886e537d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:34.128547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-500eda30-df5e-4944-b65b-7d30886e537d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4084d3ae34374a24be301694bd6c4f25', 'request_id': 'req-500eda30-df5e-4944-b65b-7d30886e537d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:34.147797 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-500eda30-df5e-4944-b65b-7d30886e537d admin admin] The token cache is being invalidate because project 4084d3ae34374a24be301694bd6c4f25 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:25:34.148093 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-500eda30-df5e-4944-b65b-7d30886e537d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:34.149069 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-500eda30-df5e-4944-b65b-7d30886e537d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:34.150621 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1278/2542] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:25:33 2022] DELETE /identity/v3/projects/4084d3ae34374a24be301694bd6c4f25 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:34.155731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6984881d-097c-4987-b083-12d8d4acf8a0 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:34.156190 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6984881d-097c-4987-b083-12d8d4acf8a0 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:34.156286 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6984881d-097c-4987-b083-12d8d4acf8a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:34.177167 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-6984881d-097c-4987-b083-12d8d4acf8a0 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:34.179020 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-6984881d-097c-4987-b083-12d8d4acf8a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:34.242382 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1265/2543] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:25:34.250629 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c04c8e4-3786-4d1c-9c86-9242c9abc74b None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:34.250954 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c04c8e4-3786-4d1c-9c86-9242c9abc74b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:34.251232 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c04c8e4-3786-4d1c-9c86-9242c9abc74b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:34.271461 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-2c04c8e4-3786-4d1c-9c86-9242c9abc74b None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:34.272737 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2c04c8e4-3786-4d1c-9c86-9242c9abc74b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:34.281288 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1279/2544] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:25:34.285876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6984881d-097c-4987-b083-12d8d4acf8a0 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:34.296641 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6984881d-097c-4987-b083-12d8d4acf8a0 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:34.301973 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565a1c79-2e8e-4c07-b7e8-ddd7a0bd6a21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:34.302765 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565a1c79-2e8e-4c07-b7e8-ddd7a0bd6a21 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:34.303139 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565a1c79-2e8e-4c07-b7e8-ddd7a0bd6a21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:34.304020 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565a1c79-2e8e-4c07-b7e8-ddd7a0bd6a21 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:34.304688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565a1c79-2e8e-4c07-b7e8-ddd7a0bd6a21 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:25:34.304994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565a1c79-2e8e-4c07-b7e8-ddd7a0bd6a21 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:34.307043 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565a1c79-2e8e-4c07-b7e8-ddd7a0bd6a21 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:34.319179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1266/2545] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:25:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:25:34.323581 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c04c8e4-3786-4d1c-9c86-9242c9abc74b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:34.333870 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c04c8e4-3786-4d1c-9c86-9242c9abc74b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:34.339304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3b8217-95b8-4b9d-a59e-59230b7f6ed1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:34.339910 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3b8217-95b8-4b9d-a59e-59230b7f6ed1 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:34.340188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3b8217-95b8-4b9d-a59e-59230b7f6ed1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:34.340457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3b8217-95b8-4b9d-a59e-59230b7f6ed1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:34.341177 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3b8217-95b8-4b9d-a59e-59230b7f6ed1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:34.343141 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3b8217-95b8-4b9d-a59e-59230b7f6ed1 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:34.372617 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1280/2546] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:25:34 2022] POST /identity/v3/projects => generated 387 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:34.376811 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565a1c79-2e8e-4c07-b7e8-ddd7a0bd6a21 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:34.386518 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565a1c79-2e8e-4c07-b7e8-ddd7a0bd6a21 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:34.444428 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16f12d2-5b36-435d-b196-9c3db347073c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:34.445029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16f12d2-5b36-435d-b196-9c3db347073c admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:34.445307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16f12d2-5b36-435d-b196-9c3db347073c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:34.446355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16f12d2-5b36-435d-b196-9c3db347073c admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:34.447120 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16f12d2-5b36-435d-b196-9c3db347073c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:34.449050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16f12d2-5b36-435d-b196-9c3db347073c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:34.470297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1267/2547] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:25:34 2022] POST /identity/v3/users => generated 294 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:34.474551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3b8217-95b8-4b9d-a59e-59230b7f6ed1 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:34.485480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3b8217-95b8-4b9d-a59e-59230b7f6ed1 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:34.490021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8df714b-767b-42da-b9a8-3a27de81958b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:34.490645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8df714b-767b-42da-b9a8-3a27de81958b admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:34.490955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8df714b-767b-42da-b9a8-3a27de81958b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:34.491231 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8df714b-767b-42da-b9a8-3a27de81958b admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:34.491859 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8df714b-767b-42da-b9a8-3a27de81958b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:34.493437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8df714b-767b-42da-b9a8-3a27de81958b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:34.503252 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1281/2548] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:34 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:34.507094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16f12d2-5b36-435d-b196-9c3db347073c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:34.516875 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16f12d2-5b36-435d-b196-9c3db347073c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:34.521459 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c33fc7d-f1ea-4fa4-85b2-642f8345f288 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:34.521983 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c33fc7d-f1ea-4fa4-85b2-642f8345f288 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:34.522925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c33fc7d-f1ea-4fa4-85b2-642f8345f288 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:34.523398 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c33fc7d-f1ea-4fa4-85b2-642f8345f288 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:34.524388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c33fc7d-f1ea-4fa4-85b2-642f8345f288 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:34.526313 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c33fc7d-f1ea-4fa4-85b2-642f8345f288 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:34.536863 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1268/2549] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:34 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:34.541610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8df714b-767b-42da-b9a8-3a27de81958b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:34.551566 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8df714b-767b-42da-b9a8-3a27de81958b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:34.556419 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c96641f-4b69-4e3c-8404-bbbb7da914bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:34.557101 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c96641f-4b69-4e3c-8404-bbbb7da914bf admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:34.557372 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c96641f-4b69-4e3c-8404-bbbb7da914bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:34.557712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c96641f-4b69-4e3c-8404-bbbb7da914bf admin admin] PATH_INFO: `/v3/projects/d28d14062fd04c1397f04401cd56ef31/users/789ac344d2ee4102996c2cd24c579a99/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:34.572330 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c96641f-4b69-4e3c-8404-bbbb7da914bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=d28d14062fd04c1397f04401cd56ef31, user_id=789ac344d2ee4102996c2cd24c579a99, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:34.575016 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c96641f-4b69-4e3c-8404-bbbb7da914bf admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:34.587839 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1282/2550] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:34 2022] PUT /identity/v3/projects/d28d14062fd04c1397f04401cd56ef31/users/789ac344d2ee4102996c2cd24c579a99/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:34.599166 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef776751-6810-4020-ae89-3766bb966f2b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:34.599457 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef776751-6810-4020-ae89-3766bb966f2b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:34.599730 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef776751-6810-4020-ae89-3766bb966f2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:34.629581 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ef776751-6810-4020-ae89-3766bb966f2b None None] MFA Rules not processed for user `789ac344d2ee4102996c2cd24c579a99`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:34.630806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ef776751-6810-4020-ae89-3766bb966f2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:34.654268 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:34.697454 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1269/2551] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:34 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:34.701929 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c96641f-4b69-4e3c-8404-bbbb7da914bf admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:34.747791 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:34.766381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c96641f-4b69-4e3c-8404-bbbb7da914bf admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:34.770978 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cb383b-9523-4f33-8fd5-e0d9b74c3611 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:34.771629 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cb383b-9523-4f33-8fd5-e0d9b74c3611 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:34.771858 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cb383b-9523-4f33-8fd5-e0d9b74c3611 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:34.772128 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cb383b-9523-4f33-8fd5-e0d9b74c3611 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:34.772741 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90cb383b-9523-4f33-8fd5-e0d9b74c3611 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:34.774611 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90cb383b-9523-4f33-8fd5-e0d9b74c3611 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:34.806022 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1283/2552] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:25:34 2022] POST /identity/v3/projects => generated 387 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:34.810093 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef776751-6810-4020-ae89-3766bb966f2b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:34.823244 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef776751-6810-4020-ae89-3766bb966f2b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:34.882745 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88f0142-e1d4-4287-a400-7a26842ece1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:34.883464 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88f0142-e1d4-4287-a400-7a26842ece1a admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:34.883738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88f0142-e1d4-4287-a400-7a26842ece1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:34.884006 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88f0142-e1d4-4287-a400-7a26842ece1a admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:34.884629 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88f0142-e1d4-4287-a400-7a26842ece1a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:34.886509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88f0142-e1d4-4287-a400-7a26842ece1a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:34.907366 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1270/2553] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:25:34 2022] POST /identity/v3/users => generated 300 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:25:34.911300 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cb383b-9523-4f33-8fd5-e0d9b74c3611 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:34.921947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cb383b-9523-4f33-8fd5-e0d9b74c3611 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:34.926543 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6a8d76-c558-4113-94a9-0215c1ab1253 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:34.927180 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6a8d76-c558-4113-94a9-0215c1ab1253 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:34.927459 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6a8d76-c558-4113-94a9-0215c1ab1253 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:34.927732 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6a8d76-c558-4113-94a9-0215c1ab1253 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:34.928992 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6a8d76-c558-4113-94a9-0215c1ab1253 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:34.930889 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6a8d76-c558-4113-94a9-0215c1ab1253 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:34.941286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1284/2554] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:34 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:34.945419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88f0142-e1d4-4287-a400-7a26842ece1a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:34.955475 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88f0142-e1d4-4287-a400-7a26842ece1a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:34.960091 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf22810-40f8-4cdf-87e2-025733916e68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:34.960744 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf22810-40f8-4cdf-87e2-025733916e68 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:34.961026 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf22810-40f8-4cdf-87e2-025733916e68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:34.961303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf22810-40f8-4cdf-87e2-025733916e68 admin admin] PATH_INFO: `/v3/projects/92a16e93d03a44f69f2619e2a646827b/users/99b3c7c65cc2493fbc94a11e224f784a/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:34.976291 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf22810-40f8-4cdf-87e2-025733916e68 admin admin] RBAC: Authorizing `identity:create_grant(project_id=92a16e93d03a44f69f2619e2a646827b, user_id=99b3c7c65cc2493fbc94a11e224f784a, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:34.978965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf22810-40f8-4cdf-87e2-025733916e68 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:34.992087 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1271/2555] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:34 2022] PUT /identity/v3/projects/92a16e93d03a44f69f2619e2a646827b/users/99b3c7c65cc2493fbc94a11e224f784a/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:34.996063 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6a8d76-c558-4113-94a9-0215c1ab1253 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:35.005838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6a8d76-c558-4113-94a9-0215c1ab1253 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:35.061393 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5003ae-342e-4843-b0c0-3614a4547308 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:35.061913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5003ae-342e-4843-b0c0-3614a4547308 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:35.062210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5003ae-342e-4843-b0c0-3614a4547308 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:35.062482 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5003ae-342e-4843-b0c0-3614a4547308 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:35.063138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5003ae-342e-4843-b0c0-3614a4547308 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:35.064710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5003ae-342e-4843-b0c0-3614a4547308 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:35.074204 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1285/2556] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:34 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:35.078349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf22810-40f8-4cdf-87e2-025733916e68 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:35.088421 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf22810-40f8-4cdf-87e2-025733916e68 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:35.094459 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af21abd0-0a2e-4b18-846a-7239f02ea9b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:35.094706 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af21abd0-0a2e-4b18-846a-7239f02ea9b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:35.095018 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af21abd0-0a2e-4b18-846a-7239f02ea9b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:35.095294 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af21abd0-0a2e-4b18-846a-7239f02ea9b4 admin admin] PATH_INFO: `/v3/projects/92a16e93d03a44f69f2619e2a646827b/users/99b3c7c65cc2493fbc94a11e224f784a/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:35.097248 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af21abd0-0a2e-4b18-846a-7239f02ea9b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=92a16e93d03a44f69f2619e2a646827b, user_id=99b3c7c65cc2493fbc94a11e224f784a, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:35.099971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af21abd0-0a2e-4b18-846a-7239f02ea9b4 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:35.112598 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1272/2557] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:35 2022] PUT /identity/v3/projects/92a16e93d03a44f69f2619e2a646827b/users/99b3c7c65cc2493fbc94a11e224f784a/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:35.123138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17c66e2-9d4f-4bbc-9e3c-30b724296a27 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:35.123522 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17c66e2-9d4f-4bbc-9e3c-30b724296a27 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:35.123882 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17c66e2-9d4f-4bbc-9e3c-30b724296a27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:35.152155 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e17c66e2-9d4f-4bbc-9e3c-30b724296a27 None None] MFA Rules not processed for user `99b3c7c65cc2493fbc94a11e224f784a`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:35.153382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e17c66e2-9d4f-4bbc-9e3c-30b724296a27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:35.251002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1286/2558] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:35 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:25:35.262018 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af21abd0-0a2e-4b18-846a-7239f02ea9b4 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:35.264010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-af21abd0-0a2e-4b18-846a-7239f02ea9b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:35.347361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af21abd0-0a2e-4b18-846a-7239f02ea9b4 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:35.353019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7978d67-be0d-474b-9b80-ebb79bfba85a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:35.353705 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7978d67-be0d-474b-9b80-ebb79bfba85a service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:35.354003 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7978d67-be0d-474b-9b80-ebb79bfba85a service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:35.354282 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7978d67-be0d-474b-9b80-ebb79bfba85a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:35.370367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7978d67-be0d-474b-9b80-ebb79bfba85a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:35.372030 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7978d67-be0d-474b-9b80-ebb79bfba85a service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:35.382683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1273/2559] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:35 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:35.525818 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17c66e2-9d4f-4bbc-9e3c-30b724296a27 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:35.577791 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:35.602935 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17c66e2-9d4f-4bbc-9e3c-30b724296a27 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:35.607859 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6dcce6-de0d-44cd-8f83-4014a5cb5ad1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:35.608497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6dcce6-de0d-44cd-8f83-4014a5cb5ad1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:35.608777 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6dcce6-de0d-44cd-8f83-4014a5cb5ad1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:35.609051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6dcce6-de0d-44cd-8f83-4014a5cb5ad1 admin admin] PATH_INFO: `/v3/users/789ac344d2ee4102996c2cd24c579a99` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:35.610382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6dcce6-de0d-44cd-8f83-4014a5cb5ad1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=789ac344d2ee4102996c2cd24c579a99)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:35.612527 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6dcce6-de0d-44cd-8f83-4014a5cb5ad1 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:35.685087 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb6dcce6-de0d-44cd-8f83-4014a5cb5ad1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '789ac344d2ee4102996c2cd24c579a99'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:35.686133 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb6dcce6-de0d-44cd-8f83-4014a5cb5ad1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '789ac344d2ee4102996c2cd24c579a99'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:35.696285 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb6dcce6-de0d-44cd-8f83-4014a5cb5ad1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '789ac344d2ee4102996c2cd24c579a99'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:35.697255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb6dcce6-de0d-44cd-8f83-4014a5cb5ad1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '789ac344d2ee4102996c2cd24c579a99'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:35.704949 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7978d67-be0d-474b-9b80-ebb79bfba85a service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:35.706720 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f7978d67-be0d-474b-9b80-ebb79bfba85a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:35.713272 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-bb6dcce6-de0d-44cd-8f83-4014a5cb5ad1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '789ac344d2ee4102996c2cd24c579a99'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:35.722916 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7978d67-be0d-474b-9b80-ebb79bfba85a service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:35.726597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1287/2560] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:25:35 2022] DELETE /identity/v3/users/789ac344d2ee4102996c2cd24c579a99 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:35.727755 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a13ca3b-4353-4f96-b65e-8d9fe5fad458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:35.728377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a13ca3b-4353-4f96-b65e-8d9fe5fad458 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:35.728654 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a13ca3b-4353-4f96-b65e-8d9fe5fad458 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:35.728923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a13ca3b-4353-4f96-b65e-8d9fe5fad458 admin admin] PATH_INFO: `/v3/users/38ba0a3f69474e4ba60361b5c48f4579` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:35.730237 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a13ca3b-4353-4f96-b65e-8d9fe5fad458 admin admin] RBAC: Authorizing `identity:delete_user(user_id=38ba0a3f69474e4ba60361b5c48f4579)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:35.730999 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017c6a03-c497-4f0f-b0c7-3a38ad7d5e50 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:35.731365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017c6a03-c497-4f0f-b0c7-3a38ad7d5e50 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:35.731707 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017c6a03-c497-4f0f-b0c7-3a38ad7d5e50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:35.732176 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a13ca3b-4353-4f96-b65e-8d9fe5fad458 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:35.747185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:35.751125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-017c6a03-c497-4f0f-b0c7-3a38ad7d5e50 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:35.752486 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-017c6a03-c497-4f0f-b0c7-3a38ad7d5e50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:35.783998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5a13ca3b-4353-4f96-b65e-8d9fe5fad458 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '38ba0a3f69474e4ba60361b5c48f4579'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:35.784843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5a13ca3b-4353-4f96-b65e-8d9fe5fad458 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '38ba0a3f69474e4ba60361b5c48f4579'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:35.793604 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5a13ca3b-4353-4f96-b65e-8d9fe5fad458 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '38ba0a3f69474e4ba60361b5c48f4579'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:35.794377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5a13ca3b-4353-4f96-b65e-8d9fe5fad458 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '38ba0a3f69474e4ba60361b5c48f4579'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:35.801665 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5a13ca3b-4353-4f96-b65e-8d9fe5fad458 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '38ba0a3f69474e4ba60361b5c48f4579'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:35.815548 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1288/2561] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:25:35.819950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1274/2562] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:25:35 2022] DELETE /identity/v3/users/38ba0a3f69474e4ba60361b5c48f4579 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:35.828668 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017c6a03-c497-4f0f-b0c7-3a38ad7d5e50 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:35.830051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-017c6a03-c497-4f0f-b0c7-3a38ad7d5e50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:35.892830 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017c6a03-c497-4f0f-b0c7-3a38ad7d5e50 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:35.902707 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:35.949735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8db6c7e-8a66-4526-ab95-d669f11f93d1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:35.950338 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8db6c7e-8a66-4526-ab95-d669f11f93d1 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:35.950612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8db6c7e-8a66-4526-ab95-d669f11f93d1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:35.950907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8db6c7e-8a66-4526-ab95-d669f11f93d1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:35.996329 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a13ca3b-4353-4f96-b65e-8d9fe5fad458 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:35.998194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5a13ca3b-4353-4f96-b65e-8d9fe5fad458 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:36.022205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8db6c7e-8a66-4526-ab95-d669f11f93d1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:36.024291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8db6c7e-8a66-4526-ab95-d669f11f93d1 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:36.034721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1289/2563] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:25:36.071671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a13ca3b-4353-4f96-b65e-8d9fe5fad458 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:36.076562 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6435e6-f771-4055-9b71-bc5411395406 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:36.077209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6435e6-f771-4055-9b71-bc5411395406 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:36.077489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6435e6-f771-4055-9b71-bc5411395406 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:36.077761 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6435e6-f771-4055-9b71-bc5411395406 admin admin] PATH_INFO: `/v3/projects/99cfba5cf9d54e10ba5fc677bf713e2f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:36.079015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6435e6-f771-4055-9b71-bc5411395406 admin admin] RBAC: Authorizing `identity:delete_project(project_id=99cfba5cf9d54e10ba5fc677bf713e2f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:36.080924 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6435e6-f771-4055-9b71-bc5411395406 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:36.129558 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2e6435e6-f771-4055-9b71-bc5411395406 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '99cfba5cf9d54e10ba5fc677bf713e2f', 'request_id': 'req-2e6435e6-f771-4055-9b71-bc5411395406'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:36.136426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2e6435e6-f771-4055-9b71-bc5411395406 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '99cfba5cf9d54e10ba5fc677bf713e2f', 'request_id': 'req-2e6435e6-f771-4055-9b71-bc5411395406'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:36.142846 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2e6435e6-f771-4055-9b71-bc5411395406 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '99cfba5cf9d54e10ba5fc677bf713e2f', 'request_id': 'req-2e6435e6-f771-4055-9b71-bc5411395406'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:36.156020 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2e6435e6-f771-4055-9b71-bc5411395406 admin admin] The token cache is being invalidate because project 99cfba5cf9d54e10ba5fc677bf713e2f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:25:36.156425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2e6435e6-f771-4055-9b71-bc5411395406 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:36.157259 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-2e6435e6-f771-4055-9b71-bc5411395406 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:36.158867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1275/2564] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:25:35 2022] DELETE /identity/v3/projects/99cfba5cf9d54e10ba5fc677bf713e2f => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:36.325429 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8db6c7e-8a66-4526-ab95-d669f11f93d1 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:36.327243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b8db6c7e-8a66-4526-ab95-d669f11f93d1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:36.345126 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8db6c7e-8a66-4526-ab95-d669f11f93d1 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:36.415682 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ee319a-81c9-45ff-95e0-23fe8d10dc22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:36.416499 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ee319a-81c9-45ff-95e0-23fe8d10dc22 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:36.416676 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ee319a-81c9-45ff-95e0-23fe8d10dc22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:36.416893 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ee319a-81c9-45ff-95e0-23fe8d10dc22 admin admin] PATH_INFO: `/v3/projects/d28d14062fd04c1397f04401cd56ef31` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:36.418062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ee319a-81c9-45ff-95e0-23fe8d10dc22 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d28d14062fd04c1397f04401cd56ef31)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:36.420180 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ee319a-81c9-45ff-95e0-23fe8d10dc22 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:36.477599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-73ee319a-81c9-45ff-95e0-23fe8d10dc22 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd28d14062fd04c1397f04401cd56ef31', 'request_id': 'req-73ee319a-81c9-45ff-95e0-23fe8d10dc22'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:36.483928 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-73ee319a-81c9-45ff-95e0-23fe8d10dc22 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd28d14062fd04c1397f04401cd56ef31', 'request_id': 'req-73ee319a-81c9-45ff-95e0-23fe8d10dc22'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:36.490519 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-73ee319a-81c9-45ff-95e0-23fe8d10dc22 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd28d14062fd04c1397f04401cd56ef31', 'request_id': 'req-73ee319a-81c9-45ff-95e0-23fe8d10dc22'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:36.493733 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:36.509161 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-73ee319a-81c9-45ff-95e0-23fe8d10dc22 admin admin] The token cache is being invalidate because project d28d14062fd04c1397f04401cd56ef31 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:25:36.509161 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-73ee319a-81c9-45ff-95e0-23fe8d10dc22 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:36.509161 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-73ee319a-81c9-45ff-95e0-23fe8d10dc22 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:36.510090 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1290/2565] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:25:36 2022] DELETE /identity/v3/projects/d28d14062fd04c1397f04401cd56ef31 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:36.513512 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6435e6-f771-4055-9b71-bc5411395406 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:36.514963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2e6435e6-f771-4055-9b71-bc5411395406 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:36.516845 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ee319a-81c9-45ff-95e0-23fe8d10dc22 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:36.518169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-73ee319a-81c9-45ff-95e0-23fe8d10dc22 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:36.527878 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6435e6-f771-4055-9b71-bc5411395406 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:36.547693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:36.576964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ee319a-81c9-45ff-95e0-23fe8d10dc22 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:36.580702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29484b2c-53b1-4c27-9888-5c5e98fd37dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:36.581356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29484b2c-53b1-4c27-9888-5c5e98fd37dc service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:36.581573 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29484b2c-53b1-4c27-9888-5c5e98fd37dc service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:36.581836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29484b2c-53b1-4c27-9888-5c5e98fd37dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:36.583602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-29484b2c-53b1-4c27-9888-5c5e98fd37dc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:36.591264 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a6a6b9-32d2-4d16-87d8-4a47e9811ee6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:36.591940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a6a6b9-32d2-4d16-87d8-4a47e9811ee6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:36.592295 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a6a6b9-32d2-4d16-87d8-4a47e9811ee6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:36.592710 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a6a6b9-32d2-4d16-87d8-4a47e9811ee6 admin admin] PATH_INFO: `/v3/users/99b3c7c65cc2493fbc94a11e224f784a` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:36.593890 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a6a6b9-32d2-4d16-87d8-4a47e9811ee6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=99b3c7c65cc2493fbc94a11e224f784a)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:36.595888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a6a6b9-32d2-4d16-87d8-4a47e9811ee6 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:36.644868 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-47a6a6b9-32d2-4d16-87d8-4a47e9811ee6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '99b3c7c65cc2493fbc94a11e224f784a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:36.645513 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-47a6a6b9-32d2-4d16-87d8-4a47e9811ee6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '99b3c7c65cc2493fbc94a11e224f784a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:36.650434 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29484b2c-53b1-4c27-9888-5c5e98fd37dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:36.652081 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29484b2c-53b1-4c27-9888-5c5e98fd37dc service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:36.653357 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-47a6a6b9-32d2-4d16-87d8-4a47e9811ee6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '99b3c7c65cc2493fbc94a11e224f784a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:36.654003 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-47a6a6b9-32d2-4d16-87d8-4a47e9811ee6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '99b3c7c65cc2493fbc94a11e224f784a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:36.661207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-47a6a6b9-32d2-4d16-87d8-4a47e9811ee6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '99b3c7c65cc2493fbc94a11e224f784a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:36.662155 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1291/2566] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:25:36.680648 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1276/2567] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:25:36 2022] DELETE /identity/v3/users/99b3c7c65cc2493fbc94a11e224f784a => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:36.970197 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29484b2c-53b1-4c27-9888-5c5e98fd37dc service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:36.972087 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-29484b2c-53b1-4c27-9888-5c5e98fd37dc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:37.007402 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29484b2c-53b1-4c27-9888-5c5e98fd37dc service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:37.066697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f3cf24-2c95-4405-84cb-620b3c96a456 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:37.067509 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f3cf24-2c95-4405-84cb-620b3c96a456 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:37.067882 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f3cf24-2c95-4405-84cb-620b3c96a456 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:37.068252 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f3cf24-2c95-4405-84cb-620b3c96a456 admin admin] PATH_INFO: `/v3/projects/92a16e93d03a44f69f2619e2a646827b` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:37.069981 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f3cf24-2c95-4405-84cb-620b3c96a456 admin admin] RBAC: Authorizing `identity:delete_project(project_id=92a16e93d03a44f69f2619e2a646827b)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:37.072119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f3cf24-2c95-4405-84cb-620b3c96a456 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:37.120188 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:DELETE=4,keystone:SELECT=96 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:37.124783 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-78f3cf24-2c95-4405-84cb-620b3c96a456 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '92a16e93d03a44f69f2619e2a646827b', 'request_id': 'req-78f3cf24-2c95-4405-84cb-620b3c96a456'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:37.132385 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-78f3cf24-2c95-4405-84cb-620b3c96a456 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '92a16e93d03a44f69f2619e2a646827b', 'request_id': 'req-78f3cf24-2c95-4405-84cb-620b3c96a456'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:37.138732 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-78f3cf24-2c95-4405-84cb-620b3c96a456 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '92a16e93d03a44f69f2619e2a646827b', 'request_id': 'req-78f3cf24-2c95-4405-84cb-620b3c96a456'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:37.153646 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-78f3cf24-2c95-4405-84cb-620b3c96a456 admin admin] The token cache is being invalidate because project 92a16e93d03a44f69f2619e2a646827b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:25:37.153931 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-78f3cf24-2c95-4405-84cb-620b3c96a456 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:37.154563 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-78f3cf24-2c95-4405-84cb-620b3c96a456 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:37.156046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1292/2568] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:25:36 2022] DELETE /identity/v3/projects/92a16e93d03a44f69f2619e2a646827b => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:37.160944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4385264-b7b2-4738-9a62-2aab91608c18 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:37.161389 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4385264-b7b2-4738-9a62-2aab91608c18 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:37.161725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4385264-b7b2-4738-9a62-2aab91608c18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:37.181836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-c4385264-b7b2-4738-9a62-2aab91608c18 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:37.183059 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c4385264-b7b2-4738-9a62-2aab91608c18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:37.244129 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1277/2569] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:37.252887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8890f64-c360-4292-8e46-1321332a3638 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:37.253176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8890f64-c360-4292-8e46-1321332a3638 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:37.253458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8890f64-c360-4292-8e46-1321332a3638 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:37.274695 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a8890f64-c360-4292-8e46-1321332a3638 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:37.275942 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a8890f64-c360-4292-8e46-1321332a3638 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:37.287095 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1293/2570] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:37.291314 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4385264-b7b2-4738-9a62-2aab91608c18 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:37.301063 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4385264-b7b2-4738-9a62-2aab91608c18 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:37.306034 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2faaae5c-1f60-4671-8856-9c367669b5c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:37.306632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faaae5c-1f60-4671-8856-9c367669b5c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:37.306933 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faaae5c-1f60-4671-8856-9c367669b5c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:37.307204 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2faaae5c-1f60-4671-8856-9c367669b5c9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:37.307957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2faaae5c-1f60-4671-8856-9c367669b5c9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:25:37.308261 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2faaae5c-1f60-4671-8856-9c367669b5c9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:37.309864 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2faaae5c-1f60-4671-8856-9c367669b5c9 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:37.321924 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1278/2571] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:25:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:25:37.326228 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8890f64-c360-4292-8e46-1321332a3638 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:37.336182 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8890f64-c360-4292-8e46-1321332a3638 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:37.341109 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f45bde-9892-4be6-ba42-5ae0107b39c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:37.341738 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f45bde-9892-4be6-ba42-5ae0107b39c9 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:37.342015 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f45bde-9892-4be6-ba42-5ae0107b39c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:37.342286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f45bde-9892-4be6-ba42-5ae0107b39c9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:37.343011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f45bde-9892-4be6-ba42-5ae0107b39c9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:37.344871 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f45bde-9892-4be6-ba42-5ae0107b39c9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:37.372720 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1294/2572] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:25:37 2022] POST /identity/v3/projects => generated 379 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:25:37.376877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2faaae5c-1f60-4671-8856-9c367669b5c9 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:37.387327 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2faaae5c-1f60-4671-8856-9c367669b5c9 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:37.419079 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:37.452923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebb241d-9bb8-46f8-94ab-bda5b2839a29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:37.453542 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebb241d-9bb8-46f8-94ab-bda5b2839a29 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:37.453820 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebb241d-9bb8-46f8-94ab-bda5b2839a29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:37.454092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebb241d-9bb8-46f8-94ab-bda5b2839a29 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:37.454714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebb241d-9bb8-46f8-94ab-bda5b2839a29 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:37.456671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebb241d-9bb8-46f8-94ab-bda5b2839a29 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:37.480680 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1279/2573] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:25:37 2022] POST /identity/v3/users => generated 290 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:37.481964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f45bde-9892-4be6-ba42-5ae0107b39c9 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:37.492992 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f45bde-9892-4be6-ba42-5ae0107b39c9 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:37.498612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554148d6-0d4c-4a1d-92c6-37ca983d31ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:37.499306 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554148d6-0d4c-4a1d-92c6-37ca983d31ce admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:37.499583 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554148d6-0d4c-4a1d-92c6-37ca983d31ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:37.499894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554148d6-0d4c-4a1d-92c6-37ca983d31ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:37.500757 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554148d6-0d4c-4a1d-92c6-37ca983d31ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:37.502595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554148d6-0d4c-4a1d-92c6-37ca983d31ce admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:37.516725 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1295/2574] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:37 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:37.517909 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebb241d-9bb8-46f8-94ab-bda5b2839a29 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:37.528257 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebb241d-9bb8-46f8-94ab-bda5b2839a29 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:37.532876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1c10f6-ed91-4316-980d-289bbea6d844 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:37.533480 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1c10f6-ed91-4316-980d-289bbea6d844 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:37.533760 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1c10f6-ed91-4316-980d-289bbea6d844 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:37.534030 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1c10f6-ed91-4316-980d-289bbea6d844 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:37.534664 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1c10f6-ed91-4316-980d-289bbea6d844 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:37.536296 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1c10f6-ed91-4316-980d-289bbea6d844 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:37.548293 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1280/2575] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:37 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:37.553832 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554148d6-0d4c-4a1d-92c6-37ca983d31ce admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:37.567106 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554148d6-0d4c-4a1d-92c6-37ca983d31ce admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:37.574322 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f799f599-5f2e-413a-a09d-39734a924488 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:37.575042 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f799f599-5f2e-413a-a09d-39734a924488 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:37.575321 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f799f599-5f2e-413a-a09d-39734a924488 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:37.575594 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f799f599-5f2e-413a-a09d-39734a924488 admin admin] PATH_INFO: `/v3/projects/1bf21f86755746d8ad202816adb24ef8/users/77868947797d4be09560e15bc6a9b727/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:37.595269 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f799f599-5f2e-413a-a09d-39734a924488 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1bf21f86755746d8ad202816adb24ef8, user_id=77868947797d4be09560e15bc6a9b727, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:37.597990 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f799f599-5f2e-413a-a09d-39734a924488 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:37.615077 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1296/2576] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:37 2022] PUT /identity/v3/projects/1bf21f86755746d8ad202816adb24ef8/users/77868947797d4be09560e15bc6a9b727/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:37.625739 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794ecc88-73cf-4c8a-85fc-90e838a40886 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:37.626197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794ecc88-73cf-4c8a-85fc-90e838a40886 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:37.626563 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794ecc88-73cf-4c8a-85fc-90e838a40886 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:37.659863 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-794ecc88-73cf-4c8a-85fc-90e838a40886 None None] MFA Rules not processed for user `77868947797d4be09560e15bc6a9b727`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:37.661396 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-794ecc88-73cf-4c8a-85fc-90e838a40886 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:37.748854 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1281/2577] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:37 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:37.753374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f799f599-5f2e-413a-a09d-39734a924488 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:37.819449 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f799f599-5f2e-413a-a09d-39734a924488 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:37.824536 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a750ce5f-a8e4-4137-9b49-91d71245a9d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:37.825155 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a750ce5f-a8e4-4137-9b49-91d71245a9d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:37.825436 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a750ce5f-a8e4-4137-9b49-91d71245a9d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:37.825717 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a750ce5f-a8e4-4137-9b49-91d71245a9d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:37.826360 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a750ce5f-a8e4-4137-9b49-91d71245a9d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:37.828287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a750ce5f-a8e4-4137-9b49-91d71245a9d2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:37.857736 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1297/2578] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:25:37 2022] POST /identity/v3/projects => generated 381 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:37.861972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794ecc88-73cf-4c8a-85fc-90e838a40886 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:37.872934 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794ecc88-73cf-4c8a-85fc-90e838a40886 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:37.929888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7b0b55-1ee0-45c7-8211-edf614818793 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:37.930768 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7b0b55-1ee0-45c7-8211-edf614818793 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:37.931075 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7b0b55-1ee0-45c7-8211-edf614818793 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:37.931375 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7b0b55-1ee0-45c7-8211-edf614818793 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:37.932009 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7b0b55-1ee0-45c7-8211-edf614818793 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:37.933926 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7b0b55-1ee0-45c7-8211-edf614818793 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:37.956144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1282/2579] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:25:37 2022] POST /identity/v3/users => generated 297 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:25:37.960657 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a750ce5f-a8e4-4137-9b49-91d71245a9d2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:37.971115 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a750ce5f-a8e4-4137-9b49-91d71245a9d2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:37.975884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb07fd76-4e9c-4629-a3e5-e4b208c59db4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:37.976486 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb07fd76-4e9c-4629-a3e5-e4b208c59db4 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:37.976797 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb07fd76-4e9c-4629-a3e5-e4b208c59db4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:37.977073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb07fd76-4e9c-4629-a3e5-e4b208c59db4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:37.977708 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb07fd76-4e9c-4629-a3e5-e4b208c59db4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:37.979351 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb07fd76-4e9c-4629-a3e5-e4b208c59db4 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:37.991337 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1298/2580] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:37 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:37.995466 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7b0b55-1ee0-45c7-8211-edf614818793 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:38.006048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7b0b55-1ee0-45c7-8211-edf614818793 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:38.012216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ff1271-ed3c-4885-b493-75b6d1ca54f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:38.012993 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ff1271-ed3c-4885-b493-75b6d1ca54f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:38.013386 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ff1271-ed3c-4885-b493-75b6d1ca54f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:38.013743 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ff1271-ed3c-4885-b493-75b6d1ca54f3 admin admin] PATH_INFO: `/v3/projects/0a8a97381afd4fd48ccbf495e264d3d7/users/e065feabf4d64cd0ab9d9932ab7bb04b/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:38.030836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ff1271-ed3c-4885-b493-75b6d1ca54f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a8a97381afd4fd48ccbf495e264d3d7, user_id=e065feabf4d64cd0ab9d9932ab7bb04b, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:38.033877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ff1271-ed3c-4885-b493-75b6d1ca54f3 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:38.053318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1283/2581] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:37 2022] PUT /identity/v3/projects/0a8a97381afd4fd48ccbf495e264d3d7/users/e065feabf4d64cd0ab9d9932ab7bb04b/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:38.054469 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb07fd76-4e9c-4629-a3e5-e4b208c59db4 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:38.064835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb07fd76-4e9c-4629-a3e5-e4b208c59db4 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:38.235992 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:38.253585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86286c8a-1513-47a9-8a5b-11f7481a0629 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:38.254191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86286c8a-1513-47a9-8a5b-11f7481a0629 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:38.254467 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86286c8a-1513-47a9-8a5b-11f7481a0629 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:38.254735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86286c8a-1513-47a9-8a5b-11f7481a0629 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:38.255404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86286c8a-1513-47a9-8a5b-11f7481a0629 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:38.257627 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86286c8a-1513-47a9-8a5b-11f7481a0629 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:38.267483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1299/2582] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:38 2022] GET /identity/v3/roles => generated 1420 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:38.271667 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ff1271-ed3c-4885-b493-75b6d1ca54f3 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:38.284599 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:38.290700 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ff1271-ed3c-4885-b493-75b6d1ca54f3 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:38.298239 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c0c42e-71ce-4fab-8943-22abb424d9fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:38.299031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c0c42e-71ce-4fab-8943-22abb424d9fe admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:38.299572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c0c42e-71ce-4fab-8943-22abb424d9fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:38.299955 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c0c42e-71ce-4fab-8943-22abb424d9fe admin admin] PATH_INFO: `/v3/projects/0a8a97381afd4fd48ccbf495e264d3d7/users/e065feabf4d64cd0ab9d9932ab7bb04b/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:38.303622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c0c42e-71ce-4fab-8943-22abb424d9fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a8a97381afd4fd48ccbf495e264d3d7, user_id=e065feabf4d64cd0ab9d9932ab7bb04b, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:38.307334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c0c42e-71ce-4fab-8943-22abb424d9fe admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:38.318826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1284/2583] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:38 2022] PUT /identity/v3/projects/0a8a97381afd4fd48ccbf495e264d3d7/users/e065feabf4d64cd0ab9d9932ab7bb04b/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:38.329915 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaeb8d8b-3036-4952-863c-9b85c147c97e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:38.330319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaeb8d8b-3036-4952-863c-9b85c147c97e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:38.330987 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaeb8d8b-3036-4952-863c-9b85c147c97e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:38.359196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-eaeb8d8b-3036-4952-863c-9b85c147c97e None None] MFA Rules not processed for user `e065feabf4d64cd0ab9d9932ab7bb04b`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:38.360798 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-eaeb8d8b-3036-4952-863c-9b85c147c97e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:38.466663 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1300/2584] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:38 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:38.480344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c0c42e-71ce-4fab-8943-22abb424d9fe admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:38.482965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-17c0c42e-71ce-4fab-8943-22abb424d9fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:38.553567 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c0c42e-71ce-4fab-8943-22abb424d9fe admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:38.558376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cbd5195-fcae-4c39-bb0e-58e5f109f5bf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:38.559121 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbd5195-fcae-4c39-bb0e-58e5f109f5bf service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:38.559462 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbd5195-fcae-4c39-bb0e-58e5f109f5bf service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:38.559805 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbd5195-fcae-4c39-bb0e-58e5f109f5bf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:38.570770 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cbd5195-fcae-4c39-bb0e-58e5f109f5bf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:38.572270 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cbd5195-fcae-4c39-bb0e-58e5f109f5bf service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:38.582470 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1285/2585] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:38 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:39.022145 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaeb8d8b-3036-4952-863c-9b85c147c97e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:39.087489 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaeb8d8b-3036-4952-863c-9b85c147c97e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:39.092577 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec52bb7-4dbb-4492-8dbd-7b2f4d23d747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:39.093294 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec52bb7-4dbb-4492-8dbd-7b2f4d23d747 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:39.093640 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec52bb7-4dbb-4492-8dbd-7b2f4d23d747 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:39.093911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec52bb7-4dbb-4492-8dbd-7b2f4d23d747 admin admin] PATH_INFO: `/v3/users/77868947797d4be09560e15bc6a9b727` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:39.095051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec52bb7-4dbb-4492-8dbd-7b2f4d23d747 admin admin] RBAC: Authorizing `identity:delete_user(user_id=77868947797d4be09560e15bc6a9b727)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:39.096975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec52bb7-4dbb-4492-8dbd-7b2f4d23d747 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:39.164155 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2ec52bb7-4dbb-4492-8dbd-7b2f4d23d747 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '77868947797d4be09560e15bc6a9b727'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:39.165206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2ec52bb7-4dbb-4492-8dbd-7b2f4d23d747 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '77868947797d4be09560e15bc6a9b727'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:39.175196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2ec52bb7-4dbb-4492-8dbd-7b2f4d23d747 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '77868947797d4be09560e15bc6a9b727'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:39.176216 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2ec52bb7-4dbb-4492-8dbd-7b2f4d23d747 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '77868947797d4be09560e15bc6a9b727'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:39.188447 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:39.202974 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2ec52bb7-4dbb-4492-8dbd-7b2f4d23d747 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '77868947797d4be09560e15bc6a9b727'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:39.204380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1301/2586] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:25:39 2022] DELETE /identity/v3/users/77868947797d4be09560e15bc6a9b727 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:39.211721 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc55edd-d461-4c36-98da-a71b250532dc None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:39.212060 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc55edd-d461-4c36-98da-a71b250532dc None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:39.212340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc55edd-d461-4c36-98da-a71b250532dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:39.239207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-cdc55edd-d461-4c36-98da-a71b250532dc None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:39.240500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cdc55edd-d461-4c36-98da-a71b250532dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:39.335157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1286/2587] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:39.342845 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec52bb7-4dbb-4492-8dbd-7b2f4d23d747 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:39.347918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2ec52bb7-4dbb-4492-8dbd-7b2f4d23d747 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:39.457887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec52bb7-4dbb-4492-8dbd-7b2f4d23d747 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:39.463229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25fe5ba-b222-431b-9dd2-d4dccaf4d6f9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:39.463853 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25fe5ba-b222-431b-9dd2-d4dccaf4d6f9 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:39.464120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25fe5ba-b222-431b-9dd2-d4dccaf4d6f9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:39.464390 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25fe5ba-b222-431b-9dd2-d4dccaf4d6f9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:39.475013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25fe5ba-b222-431b-9dd2-d4dccaf4d6f9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:39.476463 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25fe5ba-b222-431b-9dd2-d4dccaf4d6f9 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:39.488540 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1302/2588] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:40.326475 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc55edd-d461-4c36-98da-a71b250532dc None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:40.328077 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cdc55edd-d461-4c36-98da-a71b250532dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:40.350451 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc55edd-d461-4c36-98da-a71b250532dc None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:40.355329 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1833a712-f5a6-42d1-b6b5-79c238702bd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:40.356193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1833a712-f5a6-42d1-b6b5-79c238702bd7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:40.356481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1833a712-f5a6-42d1-b6b5-79c238702bd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:40.356951 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1833a712-f5a6-42d1-b6b5-79c238702bd7 admin admin] PATH_INFO: `/v3/projects/1bf21f86755746d8ad202816adb24ef8` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:40.358131 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1833a712-f5a6-42d1-b6b5-79c238702bd7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1bf21f86755746d8ad202816adb24ef8)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:40.363888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1833a712-f5a6-42d1-b6b5-79c238702bd7 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:40.435824 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1833a712-f5a6-42d1-b6b5-79c238702bd7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1bf21f86755746d8ad202816adb24ef8', 'request_id': 'req-1833a712-f5a6-42d1-b6b5-79c238702bd7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:40.446364 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1833a712-f5a6-42d1-b6b5-79c238702bd7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1bf21f86755746d8ad202816adb24ef8', 'request_id': 'req-1833a712-f5a6-42d1-b6b5-79c238702bd7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:40.453525 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1833a712-f5a6-42d1-b6b5-79c238702bd7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1bf21f86755746d8ad202816adb24ef8', 'request_id': 'req-1833a712-f5a6-42d1-b6b5-79c238702bd7'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:40.471025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1833a712-f5a6-42d1-b6b5-79c238702bd7 admin admin] The token cache is being invalidate because project 1bf21f86755746d8ad202816adb24ef8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:25:40.471363 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1833a712-f5a6-42d1-b6b5-79c238702bd7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:40.472755 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-1833a712-f5a6-42d1-b6b5-79c238702bd7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:40.477371 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1287/2589] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:25:40 2022] DELETE /identity/v3/projects/1bf21f86755746d8ad202816adb24ef8 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:40.478608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25fe5ba-b222-431b-9dd2-d4dccaf4d6f9 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:40.482784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1833a712-f5a6-42d1-b6b5-79c238702bd7 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:40.486165 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1833a712-f5a6-42d1-b6b5-79c238702bd7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:40.489259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25fe5ba-b222-431b-9dd2-d4dccaf4d6f9 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:40.506238 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:40.566066 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1833a712-f5a6-42d1-b6b5-79c238702bd7 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:40.567049 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed89af61-6b53-43de-b31a-8a76a0f02d88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:40.567680 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed89af61-6b53-43de-b31a-8a76a0f02d88 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:40.567949 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed89af61-6b53-43de-b31a-8a76a0f02d88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:40.568217 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed89af61-6b53-43de-b31a-8a76a0f02d88 admin admin] PATH_INFO: `/v3/users/e065feabf4d64cd0ab9d9932ab7bb04b` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:40.569871 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed89af61-6b53-43de-b31a-8a76a0f02d88 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e065feabf4d64cd0ab9d9932ab7bb04b)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:40.571493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a2511d-fbf3-45e3-9d8f-a369911876fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:40.571778 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed89af61-6b53-43de-b31a-8a76a0f02d88 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:40.572155 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a2511d-fbf3-45e3-9d8f-a369911876fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:40.572439 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a2511d-fbf3-45e3-9d8f-a369911876fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:40.572713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a2511d-fbf3-45e3-9d8f-a369911876fa admin admin] PATH_INFO: `/v3/users/e2846ad486314958bb74af71b647c470` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:40.573988 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a2511d-fbf3-45e3-9d8f-a369911876fa admin admin] RBAC: Authorizing `identity:delete_user(user_id=e2846ad486314958bb74af71b647c470)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:40.576359 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a2511d-fbf3-45e3-9d8f-a369911876fa admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:40.627205 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ed89af61-6b53-43de-b31a-8a76a0f02d88 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e065feabf4d64cd0ab9d9932ab7bb04b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:40.627555 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-52a2511d-fbf3-45e3-9d8f-a369911876fa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e2846ad486314958bb74af71b647c470'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:40.628046 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ed89af61-6b53-43de-b31a-8a76a0f02d88 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e065feabf4d64cd0ab9d9932ab7bb04b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:40.628221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-52a2511d-fbf3-45e3-9d8f-a369911876fa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e2846ad486314958bb74af71b647c470'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:40.635666 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ed89af61-6b53-43de-b31a-8a76a0f02d88 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e065feabf4d64cd0ab9d9932ab7bb04b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:40.636268 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ed89af61-6b53-43de-b31a-8a76a0f02d88 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e065feabf4d64cd0ab9d9932ab7bb04b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:40.638362 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-52a2511d-fbf3-45e3-9d8f-a369911876fa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e2846ad486314958bb74af71b647c470'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:40.639169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-52a2511d-fbf3-45e3-9d8f-a369911876fa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e2846ad486314958bb74af71b647c470'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:40.646641 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-52a2511d-fbf3-45e3-9d8f-a369911876fa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e2846ad486314958bb74af71b647c470'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:40.655626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ed89af61-6b53-43de-b31a-8a76a0f02d88 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e065feabf4d64cd0ab9d9932ab7bb04b'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:40.666283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1288/2590] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:25:40 2022] DELETE /identity/v3/users/e2846ad486314958bb74af71b647c470 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:25:40.673694 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1303/2591] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:25:40 2022] DELETE /identity/v3/users/e065feabf4d64cd0ab9d9932ab7bb04b => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:40.961572 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a2511d-fbf3-45e3-9d8f-a369911876fa admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:40.963778 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-52a2511d-fbf3-45e3-9d8f-a369911876fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:41.035759 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a2511d-fbf3-45e3-9d8f-a369911876fa admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:41.077521 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:41.095523 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f325009-7d78-431d-86e9-4392fc130db0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:41.096158 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f325009-7d78-431d-86e9-4392fc130db0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.096548 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f325009-7d78-431d-86e9-4392fc130db0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.096824 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f325009-7d78-431d-86e9-4392fc130db0 admin admin] PATH_INFO: `/v3/projects/7a9e1eee32f04f3cabd8219d36d2c6a7` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.097947 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f325009-7d78-431d-86e9-4392fc130db0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7a9e1eee32f04f3cabd8219d36d2c6a7)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:41.100393 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f325009-7d78-431d-86e9-4392fc130db0 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:41.152368 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5f325009-7d78-431d-86e9-4392fc130db0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7a9e1eee32f04f3cabd8219d36d2c6a7', 'request_id': 'req-5f325009-7d78-431d-86e9-4392fc130db0'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:41.159821 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5f325009-7d78-431d-86e9-4392fc130db0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7a9e1eee32f04f3cabd8219d36d2c6a7', 'request_id': 'req-5f325009-7d78-431d-86e9-4392fc130db0'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:41.160516 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed89af61-6b53-43de-b31a-8a76a0f02d88 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:41.163686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ed89af61-6b53-43de-b31a-8a76a0f02d88 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:41.169831 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5f325009-7d78-431d-86e9-4392fc130db0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7a9e1eee32f04f3cabd8219d36d2c6a7', 'request_id': 'req-5f325009-7d78-431d-86e9-4392fc130db0'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:41.179134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed89af61-6b53-43de-b31a-8a76a0f02d88 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:41.185818 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:41.194288 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5f325009-7d78-431d-86e9-4392fc130db0 admin admin] The token cache is being invalidate because project 7a9e1eee32f04f3cabd8219d36d2c6a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:25:41.194591 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5f325009-7d78-431d-86e9-4392fc130db0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:41.195350 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5f325009-7d78-431d-86e9-4392fc130db0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:41.197308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1289/2592] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:25:40 2022] DELETE /identity/v3/projects/7a9e1eee32f04f3cabd8219d36d2c6a7 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:41.210782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca709131-28f5-441b-b6a8-db41b0aa19b2 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.210782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca709131-28f5-441b-b6a8-db41b0aa19b2 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.210782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca709131-28f5-441b-b6a8-db41b0aa19b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.228555 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ca709131-28f5-441b-b6a8-db41b0aa19b2 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:41.229670 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ca709131-28f5-441b-b6a8-db41b0aa19b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:41.252717 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee888dd9-7283-47a6-a170-790094d0083a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:41.253353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee888dd9-7283-47a6-a170-790094d0083a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.253618 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee888dd9-7283-47a6-a170-790094d0083a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.253891 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee888dd9-7283-47a6-a170-790094d0083a admin admin] PATH_INFO: `/v3/projects/0a8a97381afd4fd48ccbf495e264d3d7` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.255023 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee888dd9-7283-47a6-a170-790094d0083a admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a8a97381afd4fd48ccbf495e264d3d7)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:41.256895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee888dd9-7283-47a6-a170-790094d0083a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:41.290127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1290/2593] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:41.297976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb84312-d6a5-411b-a74b-ecfd56b590d5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.298386 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb84312-d6a5-411b-a74b-ecfd56b590d5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.298666 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb84312-d6a5-411b-a74b-ecfd56b590d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.305018 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ee888dd9-7283-47a6-a170-790094d0083a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a8a97381afd4fd48ccbf495e264d3d7', 'request_id': 'req-ee888dd9-7283-47a6-a170-790094d0083a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:41.311106 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ee888dd9-7283-47a6-a170-790094d0083a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a8a97381afd4fd48ccbf495e264d3d7', 'request_id': 'req-ee888dd9-7283-47a6-a170-790094d0083a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:41.317258 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ee888dd9-7283-47a6-a170-790094d0083a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a8a97381afd4fd48ccbf495e264d3d7', 'request_id': 'req-ee888dd9-7283-47a6-a170-790094d0083a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:41.318794 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-2bb84312-d6a5-411b-a74b-ecfd56b590d5 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:41.320055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2bb84312-d6a5-411b-a74b-ecfd56b590d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:41.330045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ee888dd9-7283-47a6-a170-790094d0083a admin admin] The token cache is being invalidate because project 0a8a97381afd4fd48ccbf495e264d3d7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:25:41.330276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ee888dd9-7283-47a6-a170-790094d0083a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:41.330925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ee888dd9-7283-47a6-a170-790094d0083a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:41.332402 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1304/2594] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:25:41 2022] DELETE /identity/v3/projects/0a8a97381afd4fd48ccbf495e264d3d7 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:41.337565 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78fd8ce-003f-4246-ab84-acacd7550c40 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.337850 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78fd8ce-003f-4246-ab84-acacd7550c40 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.338121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78fd8ce-003f-4246-ab84-acacd7550c40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.357768 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a78fd8ce-003f-4246-ab84-acacd7550c40 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:41.359102 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a78fd8ce-003f-4246-ab84-acacd7550c40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:41.383489 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1291/2595] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:25:41.387672 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb84312-d6a5-411b-a74b-ecfd56b590d5 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:41.396466 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb84312-d6a5-411b-a74b-ecfd56b590d5 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:41.402106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5dda37-9d0c-46e0-8520-45202ee6f216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:41.402714 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5dda37-9d0c-46e0-8520-45202ee6f216 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.403064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5dda37-9d0c-46e0-8520-45202ee6f216 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.403340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5dda37-9d0c-46e0-8520-45202ee6f216 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.403981 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5dda37-9d0c-46e0-8520-45202ee6f216 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:25:41.404288 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5dda37-9d0c-46e0-8520-45202ee6f216 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:41.405889 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5dda37-9d0c-46e0-8520-45202ee6f216 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:41.415385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:41.421528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1292/2596] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:25:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:25:41.426035 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5dda37-9d0c-46e0-8520-45202ee6f216 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:41.434616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1305/2597] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:25:41.438742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5dda37-9d0c-46e0-8520-45202ee6f216 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:41.442459 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c150f178-13cb-4786-a4d2-b2078aee471b None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.442949 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c150f178-13cb-4786-a4d2-b2078aee471b None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.443227 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c150f178-13cb-4786-a4d2-b2078aee471b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.443376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1abb5c-e97f-4ba9-934b-cfb290b78722 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:41.443995 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1abb5c-e97f-4ba9-934b-cfb290b78722 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.444269 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1abb5c-e97f-4ba9-934b-cfb290b78722 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.444538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1abb5c-e97f-4ba9-934b-cfb290b78722 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.445155 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1abb5c-e97f-4ba9-934b-cfb290b78722 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:41.447048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1abb5c-e97f-4ba9-934b-cfb290b78722 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:41.465440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c150f178-13cb-4786-a4d2-b2078aee471b None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:41.467240 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c150f178-13cb-4786-a4d2-b2078aee471b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:41.475796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1306/2598] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:25:41.479081 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1293/2599] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:25:41 2022] POST /identity/v3/projects => generated 383 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:25:41.479949 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c150f178-13cb-4786-a4d2-b2078aee471b None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:41.480406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1abb5c-e97f-4ba9-934b-cfb290b78722 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:41.488845 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c150f178-13cb-4786-a4d2-b2078aee471b None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:41.531301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:41.563564 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1abb5c-e97f-4ba9-934b-cfb290b78722 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:41.565961 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc80939-d706-4e85-a071-f2286febc0c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:41.566568 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc80939-d706-4e85-a071-f2286febc0c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.566843 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc80939-d706-4e85-a071-f2286febc0c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.567140 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc80939-d706-4e85-a071-f2286febc0c2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.567774 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc80939-d706-4e85-a071-f2286febc0c2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:25:41.568068 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc80939-d706-4e85-a071-f2286febc0c2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:41.568380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9565301a-c8d3-44de-8aed-8de660f5fec4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:41.569541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9565301a-c8d3-44de-8aed-8de660f5fec4 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.569824 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9565301a-c8d3-44de-8aed-8de660f5fec4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.570099 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9565301a-c8d3-44de-8aed-8de660f5fec4 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.570209 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc80939-d706-4e85-a071-f2286febc0c2 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:41.570751 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9565301a-c8d3-44de-8aed-8de660f5fec4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:41.572960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9565301a-c8d3-44de-8aed-8de660f5fec4 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:41.586030 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1307/2600] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:25:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:25:41.586929 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc80939-d706-4e85-a071-f2286febc0c2 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:41.596873 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc80939-d706-4e85-a071-f2286febc0c2 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:41.598038 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1294/2601] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:25:41 2022] POST /identity/v3/users => generated 292 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:41.602069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9565301a-c8d3-44de-8aed-8de660f5fec4 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:41.604693 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0ec8dc-c91e-4332-a5cf-8377d581438c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:41.605287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0ec8dc-c91e-4332-a5cf-8377d581438c admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.605568 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0ec8dc-c91e-4332-a5cf-8377d581438c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.605840 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0ec8dc-c91e-4332-a5cf-8377d581438c admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.606461 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c0ec8dc-c91e-4332-a5cf-8377d581438c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:41.608449 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c0ec8dc-c91e-4332-a5cf-8377d581438c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:41.611768 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9565301a-c8d3-44de-8aed-8de660f5fec4 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:41.616717 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643c5669-a883-47b9-8bb2-abc805d2da05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:41.617302 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643c5669-a883-47b9-8bb2-abc805d2da05 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.617581 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643c5669-a883-47b9-8bb2-abc805d2da05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.617872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643c5669-a883-47b9-8bb2-abc805d2da05 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.618496 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643c5669-a883-47b9-8bb2-abc805d2da05 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:41.620204 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643c5669-a883-47b9-8bb2-abc805d2da05 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:41.630713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1295/2602] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:41 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:25:41.635537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1308/2603] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:25:41 2022] POST /identity/v3/projects => generated 377 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:25:41.636996 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643c5669-a883-47b9-8bb2-abc805d2da05 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:41.639294 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0ec8dc-c91e-4332-a5cf-8377d581438c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:41.647884 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643c5669-a883-47b9-8bb2-abc805d2da05 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:41.703578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0ec8dc-c91e-4332-a5cf-8377d581438c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:41.705070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174e57cc-3e40-4e8f-8919-28cb19c1a098 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:41.705676 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174e57cc-3e40-4e8f-8919-28cb19c1a098 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.705954 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174e57cc-3e40-4e8f-8919-28cb19c1a098 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.706224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174e57cc-3e40-4e8f-8919-28cb19c1a098 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.706874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174e57cc-3e40-4e8f-8919-28cb19c1a098 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:41.708282 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a3d33b-0364-4bd6-95ea-595773c6d450 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:41.708548 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174e57cc-3e40-4e8f-8919-28cb19c1a098 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:41.708883 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a3d33b-0364-4bd6-95ea-595773c6d450 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.709182 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a3d33b-0364-4bd6-95ea-595773c6d450 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.710060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a3d33b-0364-4bd6-95ea-595773c6d450 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.710680 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a3d33b-0364-4bd6-95ea-595773c6d450 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:41.712590 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a3d33b-0364-4bd6-95ea-595773c6d450 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:41.719664 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1296/2604] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:41 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:41.723725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174e57cc-3e40-4e8f-8919-28cb19c1a098 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:41.733608 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174e57cc-3e40-4e8f-8919-28cb19c1a098 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:41.735673 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1309/2605] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:25:41 2022] POST /identity/v3/users => generated 289 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:25:41.738683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99382c8b-2bc7-4229-9bd7-5633297a1c80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:41.739407 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99382c8b-2bc7-4229-9bd7-5633297a1c80 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.739541 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a3d33b-0364-4bd6-95ea-595773c6d450 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:41.739797 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99382c8b-2bc7-4229-9bd7-5633297a1c80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.739969 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99382c8b-2bc7-4229-9bd7-5633297a1c80 admin admin] PATH_INFO: `/v3/projects/91791df2a5554f08bf5827f346ed5760/users/6b6f3855d9bc48a3b70e01151a36f466/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.749534 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a3d33b-0364-4bd6-95ea-595773c6d450 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:41.754178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37153b2e-2fb9-4eae-822c-2c22b485ac55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:41.754773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37153b2e-2fb9-4eae-822c-2c22b485ac55 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.755085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37153b2e-2fb9-4eae-822c-2c22b485ac55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.755357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37153b2e-2fb9-4eae-822c-2c22b485ac55 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.755994 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37153b2e-2fb9-4eae-822c-2c22b485ac55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:41.757878 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99382c8b-2bc7-4229-9bd7-5633297a1c80 admin admin] RBAC: Authorizing `identity:create_grant(project_id=91791df2a5554f08bf5827f346ed5760, user_id=6b6f3855d9bc48a3b70e01151a36f466, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:41.758108 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37153b2e-2fb9-4eae-822c-2c22b485ac55 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:41.760932 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99382c8b-2bc7-4229-9bd7-5633297a1c80 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:41.768193 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1310/2606] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:41 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:25:41.772611 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37153b2e-2fb9-4eae-822c-2c22b485ac55 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:41.774018 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1297/2607] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:41 2022] PUT /identity/v3/projects/91791df2a5554f08bf5827f346ed5760/users/6b6f3855d9bc48a3b70e01151a36f466/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:25:41.782055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37153b2e-2fb9-4eae-822c-2c22b485ac55 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:41.783737 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd78fb4a-2be9-4ec9-bb61-c4cc69d0f2f5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.784022 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd78fb4a-2be9-4ec9-bb61-c4cc69d0f2f5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.784293 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd78fb4a-2be9-4ec9-bb61-c4cc69d0f2f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.813438 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-cd78fb4a-2be9-4ec9-bb61-c4cc69d0f2f5 None None] MFA Rules not processed for user `6b6f3855d9bc48a3b70e01151a36f466`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:41.814654 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cd78fb4a-2be9-4ec9-bb61-c4cc69d0f2f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:41.835268 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:41.851378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b85e007-07e6-47e9-a063-cab6a1001e4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:41.851974 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b85e007-07e6-47e9-a063-cab6a1001e4e admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.852259 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b85e007-07e6-47e9-a063-cab6a1001e4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.852536 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b85e007-07e6-47e9-a063-cab6a1001e4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.853178 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b85e007-07e6-47e9-a063-cab6a1001e4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:41.854823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b85e007-07e6-47e9-a063-cab6a1001e4e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:41.865417 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1311/2608] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:41 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:25:41.869469 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b85e007-07e6-47e9-a063-cab6a1001e4e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:41.878841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b85e007-07e6-47e9-a063-cab6a1001e4e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:41.883971 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683553b6-0021-43f5-996d-7c7e847b2991 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:41.884599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683553b6-0021-43f5-996d-7c7e847b2991 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.884881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683553b6-0021-43f5-996d-7c7e847b2991 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.885151 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683553b6-0021-43f5-996d-7c7e847b2991 admin admin] PATH_INFO: `/v3/projects/60f9aae8b6c147a2b7686b2f0e88c3ed/users/2471881263f54f709b9a95ba7f55f100/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.888711 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1298/2609] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:41 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:25:41.899972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd78fb4a-2be9-4ec9-bb61-c4cc69d0f2f5 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:41.900985 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683553b6-0021-43f5-996d-7c7e847b2991 admin admin] RBAC: Authorizing `identity:create_grant(project_id=60f9aae8b6c147a2b7686b2f0e88c3ed, user_id=2471881263f54f709b9a95ba7f55f100, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:41.901359 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cd78fb4a-2be9-4ec9-bb61-c4cc69d0f2f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:41.903681 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683553b6-0021-43f5-996d-7c7e847b2991 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:41.906190 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:41.919941 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1312/2610] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:41 2022] PUT /identity/v3/projects/60f9aae8b6c147a2b7686b2f0e88c3ed/users/2471881263f54f709b9a95ba7f55f100/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:25:41.930698 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653f4207-db84-4df4-9f6e-cfc38358a369 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:41.931017 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653f4207-db84-4df4-9f6e-cfc38358a369 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:41.931342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653f4207-db84-4df4-9f6e-cfc38358a369 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:41.959298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-653f4207-db84-4df4-9f6e-cfc38358a369 None None] MFA Rules not processed for user `2471881263f54f709b9a95ba7f55f100`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:41.960504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-653f4207-db84-4df4-9f6e-cfc38358a369 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:41.982005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd78fb4a-2be9-4ec9-bb61-c4cc69d0f2f5 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:42.023893 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1313/2611] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:41 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:25:42.035184 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653f4207-db84-4df4-9f6e-cfc38358a369 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:42.036602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-653f4207-db84-4df4-9f6e-cfc38358a369 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:42.047377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bc72fc-f3ae-46ac-bc8c-4ecf6445d8ed service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:42.047989 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bc72fc-f3ae-46ac-bc8c-4ecf6445d8ed service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:42.048399 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bc72fc-f3ae-46ac-bc8c-4ecf6445d8ed service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:42.048676 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bc72fc-f3ae-46ac-bc8c-4ecf6445d8ed service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:42.093787 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653f4207-db84-4df4-9f6e-cfc38358a369 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:42.097373 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8c47a6-3a8c-457e-9156-7c4f7acdc811 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eef98bcf2699461d9a7f3c3bffe82f34', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:42.097965 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8c47a6-3a8c-457e-9156-7c4f7acdc811 service glance] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:42.098242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8c47a6-3a8c-457e-9156-7c4f7acdc811 service glance] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:42.098517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8c47a6-3a8c-457e-9156-7c4f7acdc811 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:42.102750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:42.107094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20bc72fc-f3ae-46ac-bc8c-4ecf6445d8ed service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:42.107686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb8c47a6-3a8c-457e-9156-7c4f7acdc811 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:42.109011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20bc72fc-f3ae-46ac-bc8c-4ecf6445d8ed service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:42.109121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb8c47a6-3a8c-457e-9156-7c4f7acdc811 service glance] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:42.117959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1299/2612] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:41 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:25:42.118358 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1314/2613] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:42 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:25:44.694096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bc72fc-f3ae-46ac-bc8c-4ecf6445d8ed service nova] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:44.696050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-20bc72fc-f3ae-46ac-bc8c-4ecf6445d8ed service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:44.787130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bc72fc-f3ae-46ac-bc8c-4ecf6445d8ed service nova] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:44.790749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5ae77f-7871-4621-b09c-099b06483b32 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:44.791404 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5ae77f-7871-4621-b09c-099b06483b32 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:44.791677 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5ae77f-7871-4621-b09c-099b06483b32 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:44.791945 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5ae77f-7871-4621-b09c-099b06483b32 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:44.793627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4f5ae77f-7871-4621-b09c-099b06483b32 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:44.858296 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5ae77f-7871-4621-b09c-099b06483b32 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:44.859793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5ae77f-7871-4621-b09c-099b06483b32 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:44.868889 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1300/2614] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:44 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:51.725493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8c47a6-3a8c-457e-9156-7c4f7acdc811 service glance] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:51.786961 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8c47a6-3a8c-457e-9156-7c4f7acdc811 service glance] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:51.791721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3074959e-2175-4da0-b008-d1f1a9fde151 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:51.792334 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3074959e-2175-4da0-b008-d1f1a9fde151 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:51.792614 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3074959e-2175-4da0-b008-d1f1a9fde151 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:51.792887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3074959e-2175-4da0-b008-d1f1a9fde151 admin admin] PATH_INFO: `/v3/users/6b6f3855d9bc48a3b70e01151a36f466` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:51.794230 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3074959e-2175-4da0-b008-d1f1a9fde151 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b6f3855d9bc48a3b70e01151a36f466)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:51.796219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3074959e-2175-4da0-b008-d1f1a9fde151 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:51.831233 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:INSERT=1,keystone:SELECT=94,keystone:DELETE=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:51.853556 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3074959e-2175-4da0-b008-d1f1a9fde151 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b6f3855d9bc48a3b70e01151a36f466'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:51.854287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3074959e-2175-4da0-b008-d1f1a9fde151 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b6f3855d9bc48a3b70e01151a36f466'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:51.861009 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3074959e-2175-4da0-b008-d1f1a9fde151 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b6f3855d9bc48a3b70e01151a36f466'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:51.861687 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3074959e-2175-4da0-b008-d1f1a9fde151 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b6f3855d9bc48a3b70e01151a36f466'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:51.874536 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-3074959e-2175-4da0-b008-d1f1a9fde151 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b6f3855d9bc48a3b70e01151a36f466'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:51.888585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1315/2615] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:25:51 2022] DELETE /identity/v3/users/6b6f3855d9bc48a3b70e01151a36f466 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:51.893129 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95038a7-177f-409e-8c38-7b9595366cc2 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:51.893601 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95038a7-177f-409e-8c38-7b9595366cc2 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:51.893712 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95038a7-177f-409e-8c38-7b9595366cc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:51.912322 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b95038a7-177f-409e-8c38-7b9595366cc2 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:51.913495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b95038a7-177f-409e-8c38-7b9595366cc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:51.972461 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1301/2616] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:51.982695 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3074959e-2175-4da0-b008-d1f1a9fde151 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:51.984347 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-3074959e-2175-4da0-b008-d1f1a9fde151 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:52.042947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3074959e-2175-4da0-b008-d1f1a9fde151 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:52.046537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97718ffc-12b7-4ff6-b14c-91333344315f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:52.047174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97718ffc-12b7-4ff6-b14c-91333344315f service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:52.047450 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97718ffc-12b7-4ff6-b14c-91333344315f service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:52.047719 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97718ffc-12b7-4ff6-b14c-91333344315f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:52.058000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97718ffc-12b7-4ff6-b14c-91333344315f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:52.059616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97718ffc-12b7-4ff6-b14c-91333344315f service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:52.069919 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1316/2617] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:52.349951 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95038a7-177f-409e-8c38-7b9595366cc2 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:52.351513 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b95038a7-177f-409e-8c38-7b9595366cc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:52.366612 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95038a7-177f-409e-8c38-7b9595366cc2 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:52.371689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43016a5d-7cf8-4c4b-af61-ae423f9a49c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:52.372510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43016a5d-7cf8-4c4b-af61-ae423f9a49c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:52.372798 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43016a5d-7cf8-4c4b-af61-ae423f9a49c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:52.373071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43016a5d-7cf8-4c4b-af61-ae423f9a49c8 admin admin] PATH_INFO: `/v3/projects/91791df2a5554f08bf5827f346ed5760` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:52.375001 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43016a5d-7cf8-4c4b-af61-ae423f9a49c8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=91791df2a5554f08bf5827f346ed5760)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:52.377205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43016a5d-7cf8-4c4b-af61-ae423f9a49c8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:52.428605 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:52.433010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-43016a5d-7cf8-4c4b-af61-ae423f9a49c8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '91791df2a5554f08bf5827f346ed5760', 'request_id': 'req-43016a5d-7cf8-4c4b-af61-ae423f9a49c8'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:52.439729 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-43016a5d-7cf8-4c4b-af61-ae423f9a49c8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '91791df2a5554f08bf5827f346ed5760', 'request_id': 'req-43016a5d-7cf8-4c4b-af61-ae423f9a49c8'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:52.446523 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-43016a5d-7cf8-4c4b-af61-ae423f9a49c8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '91791df2a5554f08bf5827f346ed5760', 'request_id': 'req-43016a5d-7cf8-4c4b-af61-ae423f9a49c8'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:52.459159 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-43016a5d-7cf8-4c4b-af61-ae423f9a49c8 admin admin] The token cache is being invalidate because project 91791df2a5554f08bf5827f346ed5760 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:25:52.459455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-43016a5d-7cf8-4c4b-af61-ae423f9a49c8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:52.460165 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-43016a5d-7cf8-4c4b-af61-ae423f9a49c8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:52.461896 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1302/2618] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:25:52 2022] DELETE /identity/v3/projects/91791df2a5554f08bf5827f346ed5760 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:52.466680 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4663a181-85f9-40f5-b24f-035d474cd718 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:52.467000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4663a181-85f9-40f5-b24f-035d474cd718 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:52.467279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4663a181-85f9-40f5-b24f-035d474cd718 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:52.486571 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4663a181-85f9-40f5-b24f-035d474cd718 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:52.488014 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4663a181-85f9-40f5-b24f-035d474cd718 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:52.547327 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1317/2619] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:52.557252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1552c38c-b470-40a4-a287-1b01b2699b24 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:52.557643 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1552c38c-b470-40a4-a287-1b01b2699b24 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:52.557930 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1552c38c-b470-40a4-a287-1b01b2699b24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:52.579344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-1552c38c-b470-40a4-a287-1b01b2699b24 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:52.580562 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1552c38c-b470-40a4-a287-1b01b2699b24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:52.588778 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1303/2620] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:52.593648 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4663a181-85f9-40f5-b24f-035d474cd718 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:52.605192 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4663a181-85f9-40f5-b24f-035d474cd718 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:52.610094 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac012cb-7887-4cb1-9efc-20e2d2356f80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:52.610728 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac012cb-7887-4cb1-9efc-20e2d2356f80 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:52.611319 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac012cb-7887-4cb1-9efc-20e2d2356f80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:52.611459 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac012cb-7887-4cb1-9efc-20e2d2356f80 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:52.612034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac012cb-7887-4cb1-9efc-20e2d2356f80 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:25:52.612400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac012cb-7887-4cb1-9efc-20e2d2356f80 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:52.614034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac012cb-7887-4cb1-9efc-20e2d2356f80 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:52.626958 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1318/2621] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:25:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:25:52.632185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1552c38c-b470-40a4-a287-1b01b2699b24 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:52.641631 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1552c38c-b470-40a4-a287-1b01b2699b24 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:52.646311 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1120fb9-cc2e-4f03-a2f5-3a4a4d27d7ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:52.646934 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1120fb9-cc2e-4f03-a2f5-3a4a4d27d7ec admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:52.647213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1120fb9-cc2e-4f03-a2f5-3a4a4d27d7ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:52.647479 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1120fb9-cc2e-4f03-a2f5-3a4a4d27d7ec admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:52.648110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1120fb9-cc2e-4f03-a2f5-3a4a4d27d7ec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:52.650051 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1120fb9-cc2e-4f03-a2f5-3a4a4d27d7ec admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:52.678786 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1304/2622] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:25:52 2022] POST /identity/v3/projects => generated 381 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:25:52.683441 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac012cb-7887-4cb1-9efc-20e2d2356f80 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:52.694111 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac012cb-7887-4cb1-9efc-20e2d2356f80 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:52.728223 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:52.757338 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46816307-391a-462c-8b0d-8e57060798f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:52.757908 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46816307-391a-462c-8b0d-8e57060798f6 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:52.758187 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46816307-391a-462c-8b0d-8e57060798f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:52.758458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46816307-391a-462c-8b0d-8e57060798f6 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:52.759161 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46816307-391a-462c-8b0d-8e57060798f6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:52.761376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46816307-391a-462c-8b0d-8e57060798f6 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:52.787395 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1319/2623] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:25:52 2022] POST /identity/v3/users => generated 291 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:52.791745 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1120fb9-cc2e-4f03-a2f5-3a4a4d27d7ec admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:52.804611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1120fb9-cc2e-4f03-a2f5-3a4a4d27d7ec admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:52.807954 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098ec602-a6e9-45ec-9bb9-57114d76ba3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:52.808512 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098ec602-a6e9-45ec-9bb9-57114d76ba3b admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:52.808782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098ec602-a6e9-45ec-9bb9-57114d76ba3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:52.809048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098ec602-a6e9-45ec-9bb9-57114d76ba3b admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:52.809671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098ec602-a6e9-45ec-9bb9-57114d76ba3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:52.811323 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098ec602-a6e9-45ec-9bb9-57114d76ba3b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:52.822231 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1305/2624] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:52 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:52.826362 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46816307-391a-462c-8b0d-8e57060798f6 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:52.838260 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46816307-391a-462c-8b0d-8e57060798f6 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:52.843316 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af90ff43-9ded-4f6c-acc9-4f4195b05fcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:52.844005 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af90ff43-9ded-4f6c-acc9-4f4195b05fcd admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:52.844293 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af90ff43-9ded-4f6c-acc9-4f4195b05fcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:52.844607 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af90ff43-9ded-4f6c-acc9-4f4195b05fcd admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:52.845277 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af90ff43-9ded-4f6c-acc9-4f4195b05fcd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:52.846969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af90ff43-9ded-4f6c-acc9-4f4195b05fcd admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:52.869462 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1320/2625] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:52 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:52.873788 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098ec602-a6e9-45ec-9bb9-57114d76ba3b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:52.888260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098ec602-a6e9-45ec-9bb9-57114d76ba3b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:52.891136 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3483ddf6-c820-4f85-a438-5874cad1f78e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:52.891797 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3483ddf6-c820-4f85-a438-5874cad1f78e admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:52.892074 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3483ddf6-c820-4f85-a438-5874cad1f78e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:52.892344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3483ddf6-c820-4f85-a438-5874cad1f78e admin admin] PATH_INFO: `/v3/projects/ae626950093b4a7caa9153d203b3b220/users/316f91a45b2f4b34bc4eb8cf4dee17e1/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:52.908600 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3483ddf6-c820-4f85-a438-5874cad1f78e admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae626950093b4a7caa9153d203b3b220, user_id=316f91a45b2f4b34bc4eb8cf4dee17e1, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:52.911367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3483ddf6-c820-4f85-a438-5874cad1f78e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:52.927054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1306/2626] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:52 2022] PUT /identity/v3/projects/ae626950093b4a7caa9153d203b3b220/users/316f91a45b2f4b34bc4eb8cf4dee17e1/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:52.936790 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6450d364-8f2f-49f8-bb5c-37939b8c258c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:52.937171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6450d364-8f2f-49f8-bb5c-37939b8c258c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:52.937510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6450d364-8f2f-49f8-bb5c-37939b8c258c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:52.969238 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-6450d364-8f2f-49f8-bb5c-37939b8c258c None None] MFA Rules not processed for user `316f91a45b2f4b34bc4eb8cf4dee17e1`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:52.969238 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6450d364-8f2f-49f8-bb5c-37939b8c258c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:53.041802 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1321/2627] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:52 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:53.047388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3483ddf6-c820-4f85-a438-5874cad1f78e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:53.121246 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3483ddf6-c820-4f85-a438-5874cad1f78e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:53.125860 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e436f6-9740-47e5-a3b6-98501d549c8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:53.126470 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e436f6-9740-47e5-a3b6-98501d549c8e admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:53.126757 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e436f6-9740-47e5-a3b6-98501d549c8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:53.127066 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e436f6-9740-47e5-a3b6-98501d549c8e admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:53.127683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e436f6-9740-47e5-a3b6-98501d549c8e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:53.129552 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e436f6-9740-47e5-a3b6-98501d549c8e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:53.157435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1307/2628] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:25:53 2022] POST /identity/v3/projects => generated 383 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:53.161755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6450d364-8f2f-49f8-bb5c-37939b8c258c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:53.173437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6450d364-8f2f-49f8-bb5c-37939b8c258c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:53.236494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98309ea-59fe-48b8-b40c-a9e870c0c2bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:53.237106 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98309ea-59fe-48b8-b40c-a9e870c0c2bd admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:53.237388 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98309ea-59fe-48b8-b40c-a9e870c0c2bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:53.237665 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98309ea-59fe-48b8-b40c-a9e870c0c2bd admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:53.238291 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98309ea-59fe-48b8-b40c-a9e870c0c2bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:53.240257 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98309ea-59fe-48b8-b40c-a9e870c0c2bd admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:53.260990 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1322/2629] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:25:53 2022] POST /identity/v3/users => generated 298 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:53.264999 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e436f6-9740-47e5-a3b6-98501d549c8e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:53.276073 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e436f6-9740-47e5-a3b6-98501d549c8e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:53.281215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec48640c-d21a-4ef8-9474-65290360085c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:53.282011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec48640c-d21a-4ef8-9474-65290360085c admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:53.282302 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec48640c-d21a-4ef8-9474-65290360085c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:53.282586 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec48640c-d21a-4ef8-9474-65290360085c admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:53.283267 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec48640c-d21a-4ef8-9474-65290360085c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:53.285206 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec48640c-d21a-4ef8-9474-65290360085c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:53.297293 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1308/2630] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:53 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:53.301573 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98309ea-59fe-48b8-b40c-a9e870c0c2bd admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:53.312433 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98309ea-59fe-48b8-b40c-a9e870c0c2bd admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:53.317815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2a2a2e-9d8f-48e3-8dd7-206521835bb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:53.318458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2a2a2e-9d8f-48e3-8dd7-206521835bb3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:53.318733 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2a2a2e-9d8f-48e3-8dd7-206521835bb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:53.319037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2a2a2e-9d8f-48e3-8dd7-206521835bb3 admin admin] PATH_INFO: `/v3/projects/989f74621da5492a89654cf6af90414d/users/1e0017b4868b468b85d694253bb4f4fa/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:53.335220 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef2a2a2e-9d8f-48e3-8dd7-206521835bb3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=989f74621da5492a89654cf6af90414d, user_id=1e0017b4868b468b85d694253bb4f4fa, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:53.338281 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef2a2a2e-9d8f-48e3-8dd7-206521835bb3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:53.351350 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1323/2631] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:53 2022] PUT /identity/v3/projects/989f74621da5492a89654cf6af90414d/users/1e0017b4868b468b85d694253bb4f4fa/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:53.355333 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec48640c-d21a-4ef8-9474-65290360085c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:53.365529 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec48640c-d21a-4ef8-9474-65290360085c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:53.406392 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:53.427392 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05760613-2786-4254-8a81-fec067cfd3c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:53.428008 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05760613-2786-4254-8a81-fec067cfd3c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:53.428292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05760613-2786-4254-8a81-fec067cfd3c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:53.428567 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05760613-2786-4254-8a81-fec067cfd3c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:53.429208 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05760613-2786-4254-8a81-fec067cfd3c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:53.431074 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05760613-2786-4254-8a81-fec067cfd3c8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:53.441104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1309/2632] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:53 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:53.445264 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2a2a2e-9d8f-48e3-8dd7-206521835bb3 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:53.455706 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2a2a2e-9d8f-48e3-8dd7-206521835bb3 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:53.461598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c245fce4-15ca-4f68-9d6a-05e345418084 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:53.462310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c245fce4-15ca-4f68-9d6a-05e345418084 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:53.462591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c245fce4-15ca-4f68-9d6a-05e345418084 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:53.462895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c245fce4-15ca-4f68-9d6a-05e345418084 admin admin] PATH_INFO: `/v3/projects/989f74621da5492a89654cf6af90414d/users/1e0017b4868b468b85d694253bb4f4fa/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:53.464832 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c245fce4-15ca-4f68-9d6a-05e345418084 admin admin] RBAC: Authorizing `identity:create_grant(project_id=989f74621da5492a89654cf6af90414d, user_id=1e0017b4868b468b85d694253bb4f4fa, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:53.467506 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c245fce4-15ca-4f68-9d6a-05e345418084 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:53.470804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:53.479021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1324/2633] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:53 2022] PUT /identity/v3/projects/989f74621da5492a89654cf6af90414d/users/1e0017b4868b468b85d694253bb4f4fa/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:53.489181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460d0dbd-961a-4fb1-a700-acb2a0b21f9c None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:53.489472 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460d0dbd-961a-4fb1-a700-acb2a0b21f9c None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:53.490252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460d0dbd-961a-4fb1-a700-acb2a0b21f9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:53.518715 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-460d0dbd-961a-4fb1-a700-acb2a0b21f9c None None] MFA Rules not processed for user `1e0017b4868b468b85d694253bb4f4fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:53.519951 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-460d0dbd-961a-4fb1-a700-acb2a0b21f9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:53.588591 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1310/2634] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:53 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:25:53.602669 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c245fce4-15ca-4f68-9d6a-05e345418084 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:53.604424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c245fce4-15ca-4f68-9d6a-05e345418084 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:53.668948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c245fce4-15ca-4f68-9d6a-05e345418084 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:53.672631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68762d6-b3c5-46d8-b40e-25f93441e887 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:53.673234 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68762d6-b3c5-46d8-b40e-25f93441e887 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:53.673514 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68762d6-b3c5-46d8-b40e-25f93441e887 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:53.673788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68762d6-b3c5-46d8-b40e-25f93441e887 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:53.683671 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68762d6-b3c5-46d8-b40e-25f93441e887 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:53.685089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68762d6-b3c5-46d8-b40e-25f93441e887 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:53.695026 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1325/2635] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:53 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:53.769186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460d0dbd-961a-4fb1-a700-acb2a0b21f9c None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:53.770594 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-460d0dbd-961a-4fb1-a700-acb2a0b21f9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:53.783657 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460d0dbd-961a-4fb1-a700-acb2a0b21f9c None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:53.787823 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26f9199-52df-4835-b564-22c1c0e0bbe2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:53.788435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26f9199-52df-4835-b564-22c1c0e0bbe2 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:53.788715 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26f9199-52df-4835-b564-22c1c0e0bbe2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:53.788992 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26f9199-52df-4835-b564-22c1c0e0bbe2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:53.862421 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26f9199-52df-4835-b564-22c1c0e0bbe2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:53.863998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26f9199-52df-4835-b564-22c1c0e0bbe2 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:53.867824 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49af548-858d-4048-a74f-0e57fbd4d70c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:53.868633 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49af548-858d-4048-a74f-0e57fbd4d70c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:53.868910 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49af548-858d-4048-a74f-0e57fbd4d70c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:53.874968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1311/2636] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:53 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:53.888667 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c49af548-858d-4048-a74f-0e57fbd4d70c None None] MFA Rules not processed for user `2842b222fe6b4821a670d13ec8febf1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:53.889864 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c49af548-858d-4048-a74f-0e57fbd4d70c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:53.972258 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1326/2637] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:53 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:25:54.171000 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26f9199-52df-4835-b564-22c1c0e0bbe2 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:54.241921 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49af548-858d-4048-a74f-0e57fbd4d70c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:54.241921 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c49af548-858d-4048-a74f-0e57fbd4d70c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:54.270358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26f9199-52df-4835-b564-22c1c0e0bbe2 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:54.277130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb7b1cc-9cc5-463a-8a7e-f0f492026930 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:54.277775 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb7b1cc-9cc5-463a-8a7e-f0f492026930 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:54.278060 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb7b1cc-9cc5-463a-8a7e-f0f492026930 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:54.278325 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb7b1cc-9cc5-463a-8a7e-f0f492026930 admin admin] PATH_INFO: `/v3/users/316f91a45b2f4b34bc4eb8cf4dee17e1` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:54.279631 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb7b1cc-9cc5-463a-8a7e-f0f492026930 admin admin] RBAC: Authorizing `identity:delete_user(user_id=316f91a45b2f4b34bc4eb8cf4dee17e1)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:54.281545 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb7b1cc-9cc5-463a-8a7e-f0f492026930 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:54.284307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:54.325423 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49af548-858d-4048-a74f-0e57fbd4d70c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:54.329584 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6342bc5-3303-4c01-9c70-0741ec5dc1df service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:54.330204 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6342bc5-3303-4c01-9c70-0741ec5dc1df service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:54.330480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6342bc5-3303-4c01-9c70-0741ec5dc1df service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:54.330777 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6342bc5-3303-4c01-9c70-0741ec5dc1df service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:54.334895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e6342bc5-3303-4c01-9c70-0741ec5dc1df service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:54.346254 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-efb7b1cc-9cc5-463a-8a7e-f0f492026930 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '316f91a45b2f4b34bc4eb8cf4dee17e1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:54.346605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6342bc5-3303-4c01-9c70-0741ec5dc1df service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:54.347736 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-efb7b1cc-9cc5-463a-8a7e-f0f492026930 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '316f91a45b2f4b34bc4eb8cf4dee17e1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:54.348226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6342bc5-3303-4c01-9c70-0741ec5dc1df service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:54.360774 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1327/2638] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:54.362444 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-efb7b1cc-9cc5-463a-8a7e-f0f492026930 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '316f91a45b2f4b34bc4eb8cf4dee17e1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:54.363464 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-efb7b1cc-9cc5-463a-8a7e-f0f492026930 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '316f91a45b2f4b34bc4eb8cf4dee17e1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:54.372421 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-efb7b1cc-9cc5-463a-8a7e-f0f492026930 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '316f91a45b2f4b34bc4eb8cf4dee17e1'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:54.396982 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1312/2639] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:25:54 2022] DELETE /identity/v3/users/316f91a45b2f4b34bc4eb8cf4dee17e1 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:54.401582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560b2e82-b74d-4b0c-b50e-c9d37bac7415 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:54.402081 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560b2e82-b74d-4b0c-b50e-c9d37bac7415 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:54.402603 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560b2e82-b74d-4b0c-b50e-c9d37bac7415 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:54.425505 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-560b2e82-b74d-4b0c-b50e-c9d37bac7415 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:54.430455 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-560b2e82-b74d-4b0c-b50e-c9d37bac7415 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:54.478426 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:54.508938 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1328/2640] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:54.520981 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb7b1cc-9cc5-463a-8a7e-f0f492026930 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:54.523099 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-efb7b1cc-9cc5-463a-8a7e-f0f492026930 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:54.599304 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb7b1cc-9cc5-463a-8a7e-f0f492026930 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:54.603571 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f0217f-7638-46bd-8bc0-cb3af00c95e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:54.604253 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f0217f-7638-46bd-8bc0-cb3af00c95e5 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:54.604605 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f0217f-7638-46bd-8bc0-cb3af00c95e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:54.604957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f0217f-7638-46bd-8bc0-cb3af00c95e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:54.614402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f0217f-7638-46bd-8bc0-cb3af00c95e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:54.616112 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f0217f-7638-46bd-8bc0-cb3af00c95e5 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:54.627311 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1313/2641] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:55.123896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560b2e82-b74d-4b0c-b50e-c9d37bac7415 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:55.124817 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-560b2e82-b74d-4b0c-b50e-c9d37bac7415 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:55.141496 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560b2e82-b74d-4b0c-b50e-c9d37bac7415 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:55.158836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09cbfac-1506-4e64-82bd-9074fe06ea9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:55.159909 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09cbfac-1506-4e64-82bd-9074fe06ea9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:55.160480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09cbfac-1506-4e64-82bd-9074fe06ea9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:55.161002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09cbfac-1506-4e64-82bd-9074fe06ea9d admin admin] PATH_INFO: `/v3/projects/ae626950093b4a7caa9153d203b3b220` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:55.162929 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09cbfac-1506-4e64-82bd-9074fe06ea9d admin admin] RBAC: Authorizing `identity:delete_project(project_id=ae626950093b4a7caa9153d203b3b220)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:55.166452 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09cbfac-1506-4e64-82bd-9074fe06ea9d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:55.221398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a09cbfac-1506-4e64-82bd-9074fe06ea9d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ae626950093b4a7caa9153d203b3b220', 'request_id': 'req-a09cbfac-1506-4e64-82bd-9074fe06ea9d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:55.229172 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a09cbfac-1506-4e64-82bd-9074fe06ea9d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ae626950093b4a7caa9153d203b3b220', 'request_id': 'req-a09cbfac-1506-4e64-82bd-9074fe06ea9d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:55.236334 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a09cbfac-1506-4e64-82bd-9074fe06ea9d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ae626950093b4a7caa9153d203b3b220', 'request_id': 'req-a09cbfac-1506-4e64-82bd-9074fe06ea9d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:55.254406 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a09cbfac-1506-4e64-82bd-9074fe06ea9d admin admin] The token cache is being invalidate because project ae626950093b4a7caa9153d203b3b220 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:25:55.254406 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a09cbfac-1506-4e64-82bd-9074fe06ea9d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:55.255887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a09cbfac-1506-4e64-82bd-9074fe06ea9d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:55.258614 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1329/2642] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:25:55 2022] DELETE /identity/v3/projects/ae626950093b4a7caa9153d203b3b220 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:25:55.262893 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f0217f-7638-46bd-8bc0-cb3af00c95e5 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:55.281734 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f0217f-7638-46bd-8bc0-cb3af00c95e5 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:55.383355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc31dac-c832-42d6-b0c9-29e1cbb4fa46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:55.384327 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc31dac-c832-42d6-b0c9-29e1cbb4fa46 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:55.384782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc31dac-c832-42d6-b0c9-29e1cbb4fa46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:55.385284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc31dac-c832-42d6-b0c9-29e1cbb4fa46 admin admin] PATH_INFO: `/v3/users/1e0017b4868b468b85d694253bb4f4fa` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:55.386683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc31dac-c832-42d6-b0c9-29e1cbb4fa46 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1e0017b4868b468b85d694253bb4f4fa)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:55.390181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc31dac-c832-42d6-b0c9-29e1cbb4fa46 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:55.437640 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:55.464613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4cc31dac-c832-42d6-b0c9-29e1cbb4fa46 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1e0017b4868b468b85d694253bb4f4fa'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:55.465273 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4cc31dac-c832-42d6-b0c9-29e1cbb4fa46 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1e0017b4868b468b85d694253bb4f4fa'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:55.476506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4cc31dac-c832-42d6-b0c9-29e1cbb4fa46 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1e0017b4868b468b85d694253bb4f4fa'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:55.477186 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4cc31dac-c832-42d6-b0c9-29e1cbb4fa46 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1e0017b4868b468b85d694253bb4f4fa'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:55.485154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4cc31dac-c832-42d6-b0c9-29e1cbb4fa46 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1e0017b4868b468b85d694253bb4f4fa'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:55.508852 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1314/2643] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:25:55 2022] DELETE /identity/v3/users/1e0017b4868b468b85d694253bb4f4fa => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:55.831668 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09cbfac-1506-4e64-82bd-9074fe06ea9d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:55.833104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a09cbfac-1506-4e64-82bd-9074fe06ea9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:55.866276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09cbfac-1506-4e64-82bd-9074fe06ea9d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:55.933294 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06a3757-f8a9-4e39-a4f2-efe6d450eae6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:55.934099 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06a3757-f8a9-4e39-a4f2-efe6d450eae6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:55.934539 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06a3757-f8a9-4e39-a4f2-efe6d450eae6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:55.934835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06a3757-f8a9-4e39-a4f2-efe6d450eae6 admin admin] PATH_INFO: `/v3/projects/989f74621da5492a89654cf6af90414d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:55.936640 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06a3757-f8a9-4e39-a4f2-efe6d450eae6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=989f74621da5492a89654cf6af90414d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:55.938642 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06a3757-f8a9-4e39-a4f2-efe6d450eae6 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:56.014039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d06a3757-f8a9-4e39-a4f2-efe6d450eae6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '989f74621da5492a89654cf6af90414d', 'request_id': 'req-d06a3757-f8a9-4e39-a4f2-efe6d450eae6'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:56.040963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d06a3757-f8a9-4e39-a4f2-efe6d450eae6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '989f74621da5492a89654cf6af90414d', 'request_id': 'req-d06a3757-f8a9-4e39-a4f2-efe6d450eae6'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:56.054495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d06a3757-f8a9-4e39-a4f2-efe6d450eae6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '989f74621da5492a89654cf6af90414d', 'request_id': 'req-d06a3757-f8a9-4e39-a4f2-efe6d450eae6'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:56.068192 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:56.073925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d06a3757-f8a9-4e39-a4f2-efe6d450eae6 admin admin] The token cache is being invalidate because project 989f74621da5492a89654cf6af90414d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:25:56.074589 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d06a3757-f8a9-4e39-a4f2-efe6d450eae6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:56.076256 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d06a3757-f8a9-4e39-a4f2-efe6d450eae6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:56.082119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1330/2644] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:25:55 2022] DELETE /identity/v3/projects/989f74621da5492a89654cf6af90414d => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:56.085201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b73fb9-acc9-48ac-9cbd-28850620e859 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:56.086044 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b73fb9-acc9-48ac-9cbd-28850620e859 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:56.086469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b73fb9-acc9-48ac-9cbd-28850620e859 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:56.247172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-27b73fb9-acc9-48ac-9cbd-28850620e859 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:56.248455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-27b73fb9-acc9-48ac-9cbd-28850620e859 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:56.319265 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1315/2645] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:56.332390 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e7cb2b-6c16-44a3-9a9b-83498a755e0f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:56.332949 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e7cb2b-6c16-44a3-9a9b-83498a755e0f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:56.332949 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e7cb2b-6c16-44a3-9a9b-83498a755e0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:56.356941 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c6e7cb2b-6c16-44a3-9a9b-83498a755e0f None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:56.358149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c6e7cb2b-6c16-44a3-9a9b-83498a755e0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:56.367859 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1331/2646] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:56.372238 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b73fb9-acc9-48ac-9cbd-28850620e859 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:56.385445 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b73fb9-acc9-48ac-9cbd-28850620e859 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:56.390588 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034003c6-4f61-4df5-a7c2-04c91a5067a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:56.391645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034003c6-4f61-4df5-a7c2-04c91a5067a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:56.391993 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034003c6-4f61-4df5-a7c2-04c91a5067a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:56.392323 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034003c6-4f61-4df5-a7c2-04c91a5067a0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:56.393044 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034003c6-4f61-4df5-a7c2-04c91a5067a0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:25:56.393757 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034003c6-4f61-4df5-a7c2-04c91a5067a0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:56.395790 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034003c6-4f61-4df5-a7c2-04c91a5067a0 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:56.415658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e7cb2b-6c16-44a3-9a9b-83498a755e0f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:56.422870 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1316/2647] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:25:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:25:56.425321 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e7cb2b-6c16-44a3-9a9b-83498a755e0f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:56.435105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35495b46-0c97-4c5a-b397-dc5853e0e37f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:56.435710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35495b46-0c97-4c5a-b397-dc5853e0e37f admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:56.435986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35495b46-0c97-4c5a-b397-dc5853e0e37f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:56.436252 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35495b46-0c97-4c5a-b397-dc5853e0e37f admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:56.437043 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35495b46-0c97-4c5a-b397-dc5853e0e37f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:56.438981 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35495b46-0c97-4c5a-b397-dc5853e0e37f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:56.466867 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1332/2648] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:25:56 2022] POST /identity/v3/projects => generated 373 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:56.471092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034003c6-4f61-4df5-a7c2-04c91a5067a0 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:56.481335 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034003c6-4f61-4df5-a7c2-04c91a5067a0 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:56.540563 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f74910f-92a4-439d-b88e-e695375b2ad1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:56.541292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f74910f-92a4-439d-b88e-e695375b2ad1 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:56.541718 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f74910f-92a4-439d-b88e-e695375b2ad1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:56.542269 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f74910f-92a4-439d-b88e-e695375b2ad1 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:56.543002 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f74910f-92a4-439d-b88e-e695375b2ad1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:56.545437 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f74910f-92a4-439d-b88e-e695375b2ad1 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:56.569375 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1317/2649] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:25:56 2022] POST /identity/v3/users => generated 287 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:56.573408 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35495b46-0c97-4c5a-b397-dc5853e0e37f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:56.584215 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35495b46-0c97-4c5a-b397-dc5853e0e37f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:56.589008 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7676c9a4-a295-4326-bd5c-b07f446b5b28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:56.589608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7676c9a4-a295-4326-bd5c-b07f446b5b28 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:56.589915 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7676c9a4-a295-4326-bd5c-b07f446b5b28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:56.590271 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7676c9a4-a295-4326-bd5c-b07f446b5b28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:56.591074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7676c9a4-a295-4326-bd5c-b07f446b5b28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:56.592502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7676c9a4-a295-4326-bd5c-b07f446b5b28 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:56.603311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1333/2650] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:56.607545 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f74910f-92a4-439d-b88e-e695375b2ad1 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:56.619360 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f74910f-92a4-439d-b88e-e695375b2ad1 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:56.633956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1367573b-186a-4a24-b847-a6160de3a1ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:56.634753 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1367573b-186a-4a24-b847-a6160de3a1ea admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:56.634960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1367573b-186a-4a24-b847-a6160de3a1ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:56.635215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1367573b-186a-4a24-b847-a6160de3a1ea admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:56.635862 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1367573b-186a-4a24-b847-a6160de3a1ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:56.637967 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1367573b-186a-4a24-b847-a6160de3a1ea admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:56.660370 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1318/2651] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:56 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:56.664607 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7676c9a4-a295-4326-bd5c-b07f446b5b28 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:56.685396 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7676c9a4-a295-4326-bd5c-b07f446b5b28 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:56.692438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa96e56f-f9ab-422c-90bf-6017344461f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:56.693877 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa96e56f-f9ab-422c-90bf-6017344461f0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:56.695504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa96e56f-f9ab-422c-90bf-6017344461f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:56.695929 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa96e56f-f9ab-422c-90bf-6017344461f0 admin admin] PATH_INFO: `/v3/projects/5b9f4c16910f4484af30b5757bbc27f2/users/b64c3c41193f44aca60d1622afaa0175/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:56.716058 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa96e56f-f9ab-422c-90bf-6017344461f0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5b9f4c16910f4484af30b5757bbc27f2, user_id=b64c3c41193f44aca60d1622afaa0175, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:56.720675 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa96e56f-f9ab-422c-90bf-6017344461f0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:56.745525 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1334/2652] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:56 2022] PUT /identity/v3/projects/5b9f4c16910f4484af30b5757bbc27f2/users/b64c3c41193f44aca60d1622afaa0175/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:56.750675 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7109b19d-b05f-4cf8-9e4f-8c482d774ec7 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:56.751097 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7109b19d-b05f-4cf8-9e4f-8c482d774ec7 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:56.751395 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7109b19d-b05f-4cf8-9e4f-8c482d774ec7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:56.777574 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-7109b19d-b05f-4cf8-9e4f-8c482d774ec7 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:56.779094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7109b19d-b05f-4cf8-9e4f-8c482d774ec7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:56.784115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:56.908589 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1319/2653] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:56.923883 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa96e56f-f9ab-422c-90bf-6017344461f0 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:56.926999 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-fa96e56f-f9ab-422c-90bf-6017344461f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:57.048846 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa96e56f-f9ab-422c-90bf-6017344461f0 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:57.063653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6803747-b5bf-47a4-ab55-283ef2f702ec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:57.064572 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6803747-b5bf-47a4-ab55-283ef2f702ec service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:57.064921 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6803747-b5bf-47a4-ab55-283ef2f702ec service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:57.065619 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6803747-b5bf-47a4-ab55-283ef2f702ec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:57.076495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6803747-b5bf-47a4-ab55-283ef2f702ec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:57.077938 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6803747-b5bf-47a4-ab55-283ef2f702ec service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:57.089239 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1335/2654] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:25:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:25:58.486056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac48f70-2c32-40f5-bbbf-580fee9e192b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:58.492220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac48f70-2c32-40f5-bbbf-580fee9e192b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:58.492576 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac48f70-2c32-40f5-bbbf-580fee9e192b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:58.556742 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-4ac48f70-2c32-40f5-bbbf-580fee9e192b None None] MFA Rules not processed for user `b64c3c41193f44aca60d1622afaa0175`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:58.558120 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4ac48f70-2c32-40f5-bbbf-580fee9e192b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:58.694905 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1320/2655] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:58 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:58.697088 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6803747-b5bf-47a4-ab55-283ef2f702ec service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:58.709237 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6803747-b5bf-47a4-ab55-283ef2f702ec service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:58.716920 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da3b5f8-aec4-4e49-8541-ff5341572813 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:58.717555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da3b5f8-aec4-4e49-8541-ff5341572813 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:58.718004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da3b5f8-aec4-4e49-8541-ff5341572813 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:58.718412 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da3b5f8-aec4-4e49-8541-ff5341572813 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:58.719091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da3b5f8-aec4-4e49-8541-ff5341572813 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:58.722597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da3b5f8-aec4-4e49-8541-ff5341572813 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:58.775943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1336/2656] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:25:58 2022] POST /identity/v3/projects => generated 371 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:58.780258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac48f70-2c32-40f5-bbbf-580fee9e192b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:58.801181 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac48f70-2c32-40f5-bbbf-580fee9e192b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:58.849601 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da3b5f8-aec4-4e49-8541-ff5341572813 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:58.851599 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9da3b5f8-aec4-4e49-8541-ff5341572813 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:58.925701 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af34855e-426a-4948-b7a0-bd910330a5b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:58.926328 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af34855e-426a-4948-b7a0-bd910330a5b8 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:58.926612 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af34855e-426a-4948-b7a0-bd910330a5b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:58.926906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af34855e-426a-4948-b7a0-bd910330a5b8 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:58.928219 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af34855e-426a-4948-b7a0-bd910330a5b8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:58.930697 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af34855e-426a-4948-b7a0-bd910330a5b8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:58.937160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da3b5f8-aec4-4e49-8541-ff5341572813 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:58.942017 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270463f9-39c9-495f-ab34-ec5c7b4f5741 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:58.942803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270463f9-39c9-495f-ab34-ec5c7b4f5741 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:58.943174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270463f9-39c9-495f-ab34-ec5c7b4f5741 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:58.943474 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270463f9-39c9-495f-ab34-ec5c7b4f5741 admin admin] PATH_INFO: `/v3/users/776733af5b6a4115b2a11d67666a379d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:58.944854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270463f9-39c9-495f-ab34-ec5c7b4f5741 admin admin] RBAC: Authorizing `identity:delete_user(user_id=776733af5b6a4115b2a11d67666a379d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:58.946823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270463f9-39c9-495f-ab34-ec5c7b4f5741 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:58.959976 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1321/2657] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:25:58 2022] POST /identity/v3/users => generated 292 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:25:58.960862 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af34855e-426a-4948-b7a0-bd910330a5b8 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:58.971328 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af34855e-426a-4948-b7a0-bd910330a5b8 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:58.976385 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=3,keystone:DELETE=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:58.977005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea03df5-d18c-4cf6-909d-fea589c87f32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:58.977628 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea03df5-d18c-4cf6-909d-fea589c87f32 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:58.977907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea03df5-d18c-4cf6-909d-fea589c87f32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:58.978310 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea03df5-d18c-4cf6-909d-fea589c87f32 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:58.979299 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea03df5-d18c-4cf6-909d-fea589c87f32 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:58.981074 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea03df5-d18c-4cf6-909d-fea589c87f32 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:59.000221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1322/2658] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:58 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:59.002213 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea03df5-d18c-4cf6-909d-fea589c87f32 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:59.015494 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea03df5-d18c-4cf6-909d-fea589c87f32 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:59.021618 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2275e1e1-7762-4781-8dd3-598bd29ec262 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:59.022380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2275e1e1-7762-4781-8dd3-598bd29ec262 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:59.022831 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2275e1e1-7762-4781-8dd3-598bd29ec262 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:59.023225 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2275e1e1-7762-4781-8dd3-598bd29ec262 admin admin] PATH_INFO: `/v3/projects/6416e876babe4a30b21eaaa002360013/users/3412de595c01427486f5fc51bb2cf283/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:59.025263 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-270463f9-39c9-495f-ab34-ec5c7b4f5741 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '776733af5b6a4115b2a11d67666a379d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:59.026119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-270463f9-39c9-495f-ab34-ec5c7b4f5741 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '776733af5b6a4115b2a11d67666a379d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:59.037053 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-270463f9-39c9-495f-ab34-ec5c7b4f5741 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '776733af5b6a4115b2a11d67666a379d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:59.037719 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-270463f9-39c9-495f-ab34-ec5c7b4f5741 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '776733af5b6a4115b2a11d67666a379d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:59.047027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2275e1e1-7762-4781-8dd3-598bd29ec262 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6416e876babe4a30b21eaaa002360013, user_id=3412de595c01427486f5fc51bb2cf283, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:59.051290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2275e1e1-7762-4781-8dd3-598bd29ec262 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:59.054422 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:59.057145 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-270463f9-39c9-495f-ab34-ec5c7b4f5741 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '776733af5b6a4115b2a11d67666a379d'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:59.071870 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1323/2659] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:59 2022] PUT /identity/v3/projects/6416e876babe4a30b21eaaa002360013/users/3412de595c01427486f5fc51bb2cf283/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:59.075596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2275e1e1-7762-4781-8dd3-598bd29ec262 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:59.077399 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2275e1e1-7762-4781-8dd3-598bd29ec262 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:59.080807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1337/2660] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:25:58 2022] DELETE /identity/v3/users/776733af5b6a4115b2a11d67666a379d => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:25:59.153283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2275e1e1-7762-4781-8dd3-598bd29ec262 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:59.217843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a4e89c-53bb-43e3-80c5-702e42a17488 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:59.218459 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a4e89c-53bb-43e3-80c5-702e42a17488 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:59.218737 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a4e89c-53bb-43e3-80c5-702e42a17488 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:59.219048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a4e89c-53bb-43e3-80c5-702e42a17488 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:59.219691 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a4e89c-53bb-43e3-80c5-702e42a17488 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:59.221294 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a4e89c-53bb-43e3-80c5-702e42a17488 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:59.232138 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1324/2661] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:59 2022] GET /identity/v3/roles => generated 1420 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:25:59.236687 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270463f9-39c9-495f-ab34-ec5c7b4f5741 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:59.249908 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270463f9-39c9-495f-ab34-ec5c7b4f5741 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:59.255157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6212cad2-cf75-4681-8c22-8afb6fd2f351 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:59.255802 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6212cad2-cf75-4681-8c22-8afb6fd2f351 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:59.256078 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6212cad2-cf75-4681-8c22-8afb6fd2f351 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:59.256348 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6212cad2-cf75-4681-8c22-8afb6fd2f351 admin admin] PATH_INFO: `/v3/projects/6416e876babe4a30b21eaaa002360013/users/3412de595c01427486f5fc51bb2cf283/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:59.261224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a4e89c-53bb-43e3-80c5-702e42a17488 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:59.261404 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6212cad2-cf75-4681-8c22-8afb6fd2f351 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6416e876babe4a30b21eaaa002360013, user_id=3412de595c01427486f5fc51bb2cf283, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:59.263033 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-69a4e89c-53bb-43e3-80c5-702e42a17488 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:59.264175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6212cad2-cf75-4681-8c22-8afb6fd2f351 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:59.276390 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1338/2662] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:59 2022] PUT /identity/v3/projects/6416e876babe4a30b21eaaa002360013/users/3412de595c01427486f5fc51bb2cf283/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:25:59.280662 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a4e89c-53bb-43e3-80c5-702e42a17488 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:59.344140 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c65ae9-569b-430e-aac3-4b192bbf3a95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:59.344826 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c65ae9-569b-430e-aac3-4b192bbf3a95 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:59.345110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c65ae9-569b-430e-aac3-4b192bbf3a95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:59.345387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c65ae9-569b-430e-aac3-4b192bbf3a95 admin admin] PATH_INFO: `/v3/projects/30ce8239a9b34af685743812ce53ef19` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:59.346545 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c65ae9-569b-430e-aac3-4b192bbf3a95 admin admin] RBAC: Authorizing `identity:delete_project(project_id=30ce8239a9b34af685743812ce53ef19)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:59.348569 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c65ae9-569b-430e-aac3-4b192bbf3a95 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:59.391725 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:INSERT=1,keystone:SELECT=96,keystone:DELETE=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:59.400935 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-29c65ae9-569b-430e-aac3-4b192bbf3a95 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '30ce8239a9b34af685743812ce53ef19', 'request_id': 'req-29c65ae9-569b-430e-aac3-4b192bbf3a95'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:59.407868 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-29c65ae9-569b-430e-aac3-4b192bbf3a95 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '30ce8239a9b34af685743812ce53ef19', 'request_id': 'req-29c65ae9-569b-430e-aac3-4b192bbf3a95'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:59.416658 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-29c65ae9-569b-430e-aac3-4b192bbf3a95 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '30ce8239a9b34af685743812ce53ef19', 'request_id': 'req-29c65ae9-569b-430e-aac3-4b192bbf3a95'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:59.429689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-29c65ae9-569b-430e-aac3-4b192bbf3a95 admin admin] The token cache is being invalidate because project 30ce8239a9b34af685743812ce53ef19 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:25:59.430258 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-29c65ae9-569b-430e-aac3-4b192bbf3a95 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:59.430947 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-29c65ae9-569b-430e-aac3-4b192bbf3a95 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:25:59.432856 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1325/2663] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:25:59 2022] DELETE /identity/v3/projects/30ce8239a9b34af685743812ce53ef19 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:25:59.437721 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23a34bc-33ee-47d4-900e-ef05d8b0f0f3 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:59.438039 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23a34bc-33ee-47d4-900e-ef05d8b0f0f3 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:59.438309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23a34bc-33ee-47d4-900e-ef05d8b0f0f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:59.461570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c23a34bc-33ee-47d4-900e-ef05d8b0f0f3 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:59.462879 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c23a34bc-33ee-47d4-900e-ef05d8b0f0f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:59.525907 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1339/2664] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:25:59.533893 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1097c10d-26a7-4d17-9102-361eae20ad6d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:59.534424 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1097c10d-26a7-4d17-9102-361eae20ad6d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:59.534424 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1097c10d-26a7-4d17-9102-361eae20ad6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:59.556144 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-1097c10d-26a7-4d17-9102-361eae20ad6d None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:25:59.557359 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1097c10d-26a7-4d17-9102-361eae20ad6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:25:59.565601 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1326/2665] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:25:59.570203 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23a34bc-33ee-47d4-900e-ef05d8b0f0f3 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:59.580502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23a34bc-33ee-47d4-900e-ef05d8b0f0f3 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:59.586382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1539664b-1385-477a-8c49-b0c1b9cbc54a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:59.587067 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1539664b-1385-477a-8c49-b0c1b9cbc54a admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:59.587376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1539664b-1385-477a-8c49-b0c1b9cbc54a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:59.587652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1539664b-1385-477a-8c49-b0c1b9cbc54a admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:59.588301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1539664b-1385-477a-8c49-b0c1b9cbc54a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:25:59.588602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1539664b-1385-477a-8c49-b0c1b9cbc54a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:59.590182 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1539664b-1385-477a-8c49-b0c1b9cbc54a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:59.601932 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1340/2666] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:25:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:25:59.606576 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1097c10d-26a7-4d17-9102-361eae20ad6d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:59.615960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1097c10d-26a7-4d17-9102-361eae20ad6d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:59.621263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25244776-baab-4771-b5d6-daeb5f7bb5c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:59.622039 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25244776-baab-4771-b5d6-daeb5f7bb5c0 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:59.622311 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25244776-baab-4771-b5d6-daeb5f7bb5c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:59.622601 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25244776-baab-4771-b5d6-daeb5f7bb5c0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:59.623248 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25244776-baab-4771-b5d6-daeb5f7bb5c0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:59.625128 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25244776-baab-4771-b5d6-daeb5f7bb5c0 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:59.662927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1327/2667] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:25:59 2022] POST /identity/v3/projects => generated 371 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:25:59.664816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1539664b-1385-477a-8c49-b0c1b9cbc54a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:59.677233 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1539664b-1385-477a-8c49-b0c1b9cbc54a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:59.753416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a449860-fcd8-41f2-95aa-7ef3c4d7d82c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:59.754687 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a449860-fcd8-41f2-95aa-7ef3c4d7d82c admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:59.755887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a449860-fcd8-41f2-95aa-7ef3c4d7d82c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:59.756398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a449860-fcd8-41f2-95aa-7ef3c4d7d82c admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:59.757183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a449860-fcd8-41f2-95aa-7ef3c4d7d82c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:59.760285 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a449860-fcd8-41f2-95aa-7ef3c4d7d82c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:59.783300 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:DELETE=8,keystone:SELECT=87,keystone:INSERT=5 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:25:59.795310 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1341/2668] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:25:59 2022] POST /identity/v3/users => generated 286 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:25:59.798079 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25244776-baab-4771-b5d6-daeb5f7bb5c0 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:59.814027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25244776-baab-4771-b5d6-daeb5f7bb5c0 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:59.823199 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4b8c68-1cc5-4a5d-819a-d53112f10356 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:59.824054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4b8c68-1cc5-4a5d-819a-d53112f10356 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:59.824496 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4b8c68-1cc5-4a5d-819a-d53112f10356 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:59.825198 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4b8c68-1cc5-4a5d-819a-d53112f10356 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:59.826097 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4b8c68-1cc5-4a5d-819a-d53112f10356 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:59.828156 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4b8c68-1cc5-4a5d-819a-d53112f10356 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:59.845931 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1328/2669] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:59 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:25:59.846070 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a449860-fcd8-41f2-95aa-7ef3c4d7d82c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:59.865702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a449860-fcd8-41f2-95aa-7ef3c4d7d82c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:59.876351 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4705f108-40fe-4e92-a7f1-d4c0f97bb645 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:59.877440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4705f108-40fe-4e92-a7f1-d4c0f97bb645 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:59.877692 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4705f108-40fe-4e92-a7f1-d4c0f97bb645 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:59.877972 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4705f108-40fe-4e92-a7f1-d4c0f97bb645 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:59.878626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4705f108-40fe-4e92-a7f1-d4c0f97bb645 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:59.881854 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4705f108-40fe-4e92-a7f1-d4c0f97bb645 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:59.899752 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1342/2670] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:25:59 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:25:59.902118 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4b8c68-1cc5-4a5d-819a-d53112f10356 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:25:59.916409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4b8c68-1cc5-4a5d-819a-d53112f10356 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:25:59.931602 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6236004f-6c91-47df-b62f-1bdfb186d26b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:25:59.932333 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6236004f-6c91-47df-b62f-1bdfb186d26b admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:59.932618 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6236004f-6c91-47df-b62f-1bdfb186d26b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:59.932892 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6236004f-6c91-47df-b62f-1bdfb186d26b admin admin] PATH_INFO: `/v3/projects/cce68dff03de4de5be79b5425b7b8bbe/users/c2dd1fba00284664b4572d394143f81e/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:25:59.953763 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6236004f-6c91-47df-b62f-1bdfb186d26b admin admin] RBAC: Authorizing `identity:create_grant(project_id=cce68dff03de4de5be79b5425b7b8bbe, user_id=c2dd1fba00284664b4572d394143f81e, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:25:59.956538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6236004f-6c91-47df-b62f-1bdfb186d26b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:25:59.982959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1329/2671] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:25:59 2022] PUT /identity/v3/projects/cce68dff03de4de5be79b5425b7b8bbe/users/c2dd1fba00284664b4572d394143f81e/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:25:59.985714 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a6f1eb-d6ee-45ee-84fc-e461561978ab None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:25:59.986006 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a6f1eb-d6ee-45ee-84fc-e461561978ab None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:25:59.986287 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a6f1eb-d6ee-45ee-84fc-e461561978ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:00.017936 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c6a6f1eb-d6ee-45ee-84fc-e461561978ab None None] MFA Rules not processed for user `c2dd1fba00284664b4572d394143f81e`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:00.019185 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c6a6f1eb-d6ee-45ee-84fc-e461561978ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:00.101753 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1343/2672] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:25:59 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:00.102915 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6236004f-6c91-47df-b62f-1bdfb186d26b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:00.195493 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6236004f-6c91-47df-b62f-1bdfb186d26b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:00.197260 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aead04cf-8689-4cc9-bd69-95b0c10cd783 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:00.197613 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aead04cf-8689-4cc9-bd69-95b0c10cd783 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:00.197979 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aead04cf-8689-4cc9-bd69-95b0c10cd783 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:00.200882 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00161590-a1fa-4560-8d3b-09d590e38b18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:00.201903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00161590-a1fa-4560-8d3b-09d590e38b18 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:00.202388 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00161590-a1fa-4560-8d3b-09d590e38b18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:00.202681 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00161590-a1fa-4560-8d3b-09d590e38b18 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:00.203362 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00161590-a1fa-4560-8d3b-09d590e38b18 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:00.205409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00161590-a1fa-4560-8d3b-09d590e38b18 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:00.239970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-aead04cf-8689-4cc9-bd69-95b0c10cd783 None None] MFA Rules not processed for user `3412de595c01427486f5fc51bb2cf283`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:00.240873 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1330/2673] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:26:00 2022] POST /identity/v3/projects => generated 371 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:26:00.241765 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-aead04cf-8689-4cc9-bd69-95b0c10cd783 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:00.245080 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00161590-a1fa-4560-8d3b-09d590e38b18 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:00.303823 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:00.321139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1344/2674] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:00 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:26:00.322086 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00161590-a1fa-4560-8d3b-09d590e38b18 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:00.326966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3baeaac-4866-404c-aa0c-7b3417fe48e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:00.327707 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3baeaac-4866-404c-aa0c-7b3417fe48e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:00.327807 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3baeaac-4866-404c-aa0c-7b3417fe48e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:00.328020 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3baeaac-4866-404c-aa0c-7b3417fe48e4 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:00.328641 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3baeaac-4866-404c-aa0c-7b3417fe48e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:00.330532 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3baeaac-4866-404c-aa0c-7b3417fe48e4 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:00.332775 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aead04cf-8689-4cc9-bd69-95b0c10cd783 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:00.334948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-aead04cf-8689-4cc9-bd69-95b0c10cd783 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:00.355305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1331/2675] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:26:00 2022] POST /identity/v3/users => generated 292 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:26:00.359230 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3baeaac-4866-404c-aa0c-7b3417fe48e4 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:00.370321 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3baeaac-4866-404c-aa0c-7b3417fe48e4 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:00.382764 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d379439-2041-4a99-8985-a13ecb5199f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:00.383416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d379439-2041-4a99-8985-a13ecb5199f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:00.383694 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d379439-2041-4a99-8985-a13ecb5199f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:00.383964 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d379439-2041-4a99-8985-a13ecb5199f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:00.384611 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d379439-2041-4a99-8985-a13ecb5199f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:00.386624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d379439-2041-4a99-8985-a13ecb5199f4 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:00.407943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aead04cf-8689-4cc9-bd69-95b0c10cd783 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:00.408148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1332/2676] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:00 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:26:00.408253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782e485f-36b2-46f1-9b34-9b978550e8c3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:00.408253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782e485f-36b2-46f1-9b34-9b978550e8c3 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:00.408253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782e485f-36b2-46f1-9b34-9b978550e8c3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:00.408500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782e485f-36b2-46f1-9b34-9b978550e8c3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:00.408886 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d379439-2041-4a99-8985-a13ecb5199f4 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:00.410186 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-782e485f-36b2-46f1-9b34-9b978550e8c3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:00.420533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d379439-2041-4a99-8985-a13ecb5199f4 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:00.425463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a1f8df-0e82-436b-96fd-f9a9e991181b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:00.426241 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a1f8df-0e82-436b-96fd-f9a9e991181b admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:00.426586 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a1f8df-0e82-436b-96fd-f9a9e991181b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:00.427008 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a1f8df-0e82-436b-96fd-f9a9e991181b admin admin] PATH_INFO: `/v3/projects/26d09e908be540a3a7e1b7548e735766/users/1ac1bcade40a4d5f91aeb2cdb29572d2/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:00.428908 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:00.443703 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a1f8df-0e82-436b-96fd-f9a9e991181b admin admin] RBAC: Authorizing `identity:create_grant(project_id=26d09e908be540a3a7e1b7548e735766, user_id=1ac1bcade40a4d5f91aeb2cdb29572d2, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:00.446478 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a1f8df-0e82-436b-96fd-f9a9e991181b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:00.460840 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1333/2677] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:26:00 2022] PUT /identity/v3/projects/26d09e908be540a3a7e1b7548e735766/users/1ac1bcade40a4d5f91aeb2cdb29572d2/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:00.465185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a1f8df-0e82-436b-96fd-f9a9e991181b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:00.474413 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782e485f-36b2-46f1-9b34-9b978550e8c3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:00.476416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782e485f-36b2-46f1-9b34-9b978550e8c3 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:00.486034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1345/2678] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:00 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:26:00.529518 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a1f8df-0e82-436b-96fd-f9a9e991181b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:00.534246 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84790218-40d4-4744-9ff0-afb5c13c3caa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:00.534936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84790218-40d4-4744-9ff0-afb5c13c3caa admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:00.535158 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84790218-40d4-4744-9ff0-afb5c13c3caa admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:00.535451 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84790218-40d4-4744-9ff0-afb5c13c3caa admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:00.536108 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84790218-40d4-4744-9ff0-afb5c13c3caa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:00.537766 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84790218-40d4-4744-9ff0-afb5c13c3caa admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:00.548009 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1334/2679] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:00 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:26:00.552385 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782e485f-36b2-46f1-9b34-9b978550e8c3 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:00.563099 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782e485f-36b2-46f1-9b34-9b978550e8c3 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:00.568169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31533d6a-4a72-4398-9571-65d5160e0c75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:00.568825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31533d6a-4a72-4398-9571-65d5160e0c75 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:00.569106 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31533d6a-4a72-4398-9571-65d5160e0c75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:00.569382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31533d6a-4a72-4398-9571-65d5160e0c75 admin admin] PATH_INFO: `/v3/projects/26d09e908be540a3a7e1b7548e735766/users/1ac1bcade40a4d5f91aeb2cdb29572d2/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:00.571341 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31533d6a-4a72-4398-9571-65d5160e0c75 admin admin] RBAC: Authorizing `identity:create_grant(project_id=26d09e908be540a3a7e1b7548e735766, user_id=1ac1bcade40a4d5f91aeb2cdb29572d2, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:00.573949 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31533d6a-4a72-4398-9571-65d5160e0c75 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:00.587571 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1346/2680] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:26:00 2022] PUT /identity/v3/projects/26d09e908be540a3a7e1b7548e735766/users/1ac1bcade40a4d5f91aeb2cdb29572d2/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:26:00.607272 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5616dd4-f9cb-4e3e-8417-62a8af0da6e8 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:00.607695 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5616dd4-f9cb-4e3e-8417-62a8af0da6e8 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:00.608094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5616dd4-f9cb-4e3e-8417-62a8af0da6e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:00.654177 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b5616dd4-f9cb-4e3e-8417-62a8af0da6e8 None None] MFA Rules not processed for user `1ac1bcade40a4d5f91aeb2cdb29572d2`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:00.655867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b5616dd4-f9cb-4e3e-8417-62a8af0da6e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:00.743317 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1335/2681] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:00 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:00.767301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31533d6a-4a72-4398-9571-65d5160e0c75 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:00.857820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31533d6a-4a72-4398-9571-65d5160e0c75 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:00.865113 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9425110b-51c6-4a78-8fff-4bd00122aaf3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:00.865830 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9425110b-51c6-4a78-8fff-4bd00122aaf3 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:00.867262 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9425110b-51c6-4a78-8fff-4bd00122aaf3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:00.867615 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9425110b-51c6-4a78-8fff-4bd00122aaf3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:00.879504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9425110b-51c6-4a78-8fff-4bd00122aaf3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:00.881752 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9425110b-51c6-4a78-8fff-4bd00122aaf3 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:00.894032 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1347/2682] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:00 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:26:00.982953 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5616dd4-f9cb-4e3e-8417-62a8af0da6e8 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:00.995047 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5616dd4-f9cb-4e3e-8417-62a8af0da6e8 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:00.999193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f08216-a04a-4bd6-8113-d933e338829a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:00.999914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f08216-a04a-4bd6-8113-d933e338829a service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:01.000361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f08216-a04a-4bd6-8113-d933e338829a service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:01.000627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f08216-a04a-4bd6-8113-d933e338829a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:01.024484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:01.067433 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f08216-a04a-4bd6-8113-d933e338829a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:01.069140 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f08216-a04a-4bd6-8113-d933e338829a service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:01.080093 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1336/2683] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:00 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:26:01.886441 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9425110b-51c6-4a78-8fff-4bd00122aaf3 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:01.888325 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9425110b-51c6-4a78-8fff-4bd00122aaf3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:01.899503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9425110b-51c6-4a78-8fff-4bd00122aaf3 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:01.903181 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6812530-cc61-47d3-b8c2-16b1b2f6d921 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:01.903773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6812530-cc61-47d3-b8c2-16b1b2f6d921 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:01.904048 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6812530-cc61-47d3-b8c2-16b1b2f6d921 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:01.904318 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6812530-cc61-47d3-b8c2-16b1b2f6d921 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:01.966544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6812530-cc61-47d3-b8c2-16b1b2f6d921 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:01.967901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6812530-cc61-47d3-b8c2-16b1b2f6d921 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:01.978381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1348/2684] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:01 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:26:02.229406 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f08216-a04a-4bd6-8113-d933e338829a service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:02.230913 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b4f08216-a04a-4bd6-8113-d933e338829a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:02.303256 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f08216-a04a-4bd6-8113-d933e338829a service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:02.308118 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca82eed-dd2f-4baf-b274-ab56265d198e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:02.308748 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca82eed-dd2f-4baf-b274-ab56265d198e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:02.309021 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca82eed-dd2f-4baf-b274-ab56265d198e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:02.309290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca82eed-dd2f-4baf-b274-ab56265d198e admin admin] PATH_INFO: `/v3/users/2471881263f54f709b9a95ba7f55f100` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:02.310411 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca82eed-dd2f-4baf-b274-ab56265d198e admin admin] RBAC: Authorizing `identity:delete_user(user_id=2471881263f54f709b9a95ba7f55f100)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:02.312359 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca82eed-dd2f-4baf-b274-ab56265d198e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:02.366839 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bca82eed-dd2f-4baf-b274-ab56265d198e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2471881263f54f709b9a95ba7f55f100'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:02.367513 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bca82eed-dd2f-4baf-b274-ab56265d198e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2471881263f54f709b9a95ba7f55f100'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:02.374823 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bca82eed-dd2f-4baf-b274-ab56265d198e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2471881263f54f709b9a95ba7f55f100'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:02.375469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bca82eed-dd2f-4baf-b274-ab56265d198e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2471881263f54f709b9a95ba7f55f100'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:02.382506 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bca82eed-dd2f-4baf-b274-ab56265d198e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2471881263f54f709b9a95ba7f55f100'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:02.413719 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1337/2685] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:26:02 2022] DELETE /identity/v3/users/2471881263f54f709b9a95ba7f55f100 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:02.415938 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8066c952-d782-40bd-b49a-4a14cbc1cf04 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:02.416499 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8066c952-d782-40bd-b49a-4a14cbc1cf04 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:02.416937 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8066c952-d782-40bd-b49a-4a14cbc1cf04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:02.443743 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-8066c952-d782-40bd-b49a-4a14cbc1cf04 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:02.445017 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8066c952-d782-40bd-b49a-4a14cbc1cf04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:02.494617 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:02.521964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1349/2686] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:02.535191 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca82eed-dd2f-4baf-b274-ab56265d198e admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:02.538310 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-bca82eed-dd2f-4baf-b274-ab56265d198e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:02.647470 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:02.671402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca82eed-dd2f-4baf-b274-ab56265d198e admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:02.678305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6d913f-55da-4554-b727-ab811a44a93a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:02.678959 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6d913f-55da-4554-b727-ab811a44a93a service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:02.679229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6d913f-55da-4554-b727-ab811a44a93a service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:02.679499 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6d913f-55da-4554-b727-ab811a44a93a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:02.689684 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6d913f-55da-4554-b727-ab811a44a93a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:02.691398 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6d913f-55da-4554-b727-ab811a44a93a service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:02.702845 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1338/2687] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:26:02.987938 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8066c952-d782-40bd-b49a-4a14cbc1cf04 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:02.989410 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-8066c952-d782-40bd-b49a-4a14cbc1cf04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:03.001628 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8066c952-d782-40bd-b49a-4a14cbc1cf04 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:03.006268 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f048030-8c3f-4bbc-8fbc-2a3f25da89cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:03.006924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f048030-8c3f-4bbc-8fbc-2a3f25da89cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:03.007206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f048030-8c3f-4bbc-8fbc-2a3f25da89cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:03.007985 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f048030-8c3f-4bbc-8fbc-2a3f25da89cf admin admin] PATH_INFO: `/v3/projects/60f9aae8b6c147a2b7686b2f0e88c3ed` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:03.008581 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f048030-8c3f-4bbc-8fbc-2a3f25da89cf admin admin] RBAC: Authorizing `identity:delete_project(project_id=60f9aae8b6c147a2b7686b2f0e88c3ed)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:03.010658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f048030-8c3f-4bbc-8fbc-2a3f25da89cf admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:03.057761 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5f048030-8c3f-4bbc-8fbc-2a3f25da89cf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '60f9aae8b6c147a2b7686b2f0e88c3ed', 'request_id': 'req-5f048030-8c3f-4bbc-8fbc-2a3f25da89cf'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:03.063618 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5f048030-8c3f-4bbc-8fbc-2a3f25da89cf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '60f9aae8b6c147a2b7686b2f0e88c3ed', 'request_id': 'req-5f048030-8c3f-4bbc-8fbc-2a3f25da89cf'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:03.070101 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5f048030-8c3f-4bbc-8fbc-2a3f25da89cf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '60f9aae8b6c147a2b7686b2f0e88c3ed', 'request_id': 'req-5f048030-8c3f-4bbc-8fbc-2a3f25da89cf'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:03.084879 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5f048030-8c3f-4bbc-8fbc-2a3f25da89cf admin admin] The token cache is being invalidate because project 60f9aae8b6c147a2b7686b2f0e88c3ed was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:26:03.085164 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5f048030-8c3f-4bbc-8fbc-2a3f25da89cf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:03.085893 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5f048030-8c3f-4bbc-8fbc-2a3f25da89cf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:03.087747 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1350/2688] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:26:02 2022] DELETE /identity/v3/projects/60f9aae8b6c147a2b7686b2f0e88c3ed => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:03.092775 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d3e5b8-75d3-44ce-acb9-7a3e76521b41 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:03.093062 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d3e5b8-75d3-44ce-acb9-7a3e76521b41 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:03.093336 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d3e5b8-75d3-44ce-acb9-7a3e76521b41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:03.113022 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d0d3e5b8-75d3-44ce-acb9-7a3e76521b41 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:03.114209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d0d3e5b8-75d3-44ce-acb9-7a3e76521b41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:03.176420 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1339/2689] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:03.241780 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea7722d-3da8-43c5-b1b4-2b2bf9162c13 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:03.242061 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea7722d-3da8-43c5-b1b4-2b2bf9162c13 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:03.242332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea7722d-3da8-43c5-b1b4-2b2bf9162c13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:03.261548 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-aea7722d-3da8-43c5-b1b4-2b2bf9162c13 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:03.262724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-aea7722d-3da8-43c5-b1b4-2b2bf9162c13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:03.271191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1351/2690] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:03.275459 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d3e5b8-75d3-44ce-acb9-7a3e76521b41 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:03.286142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d3e5b8-75d3-44ce-acb9-7a3e76521b41 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:03.291481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ff2e57-b791-4a0c-bbd5-0fbc7468e357 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:03.292082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ff2e57-b791-4a0c-bbd5-0fbc7468e357 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:03.292387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ff2e57-b791-4a0c-bbd5-0fbc7468e357 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:03.292660 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ff2e57-b791-4a0c-bbd5-0fbc7468e357 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:03.293502 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ff2e57-b791-4a0c-bbd5-0fbc7468e357 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:26:03.293840 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ff2e57-b791-4a0c-bbd5-0fbc7468e357 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:03.295499 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ff2e57-b791-4a0c-bbd5-0fbc7468e357 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:03.312042 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1340/2691] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:26:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:26:03.313139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea7722d-3da8-43c5-b1b4-2b2bf9162c13 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:03.323153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea7722d-3da8-43c5-b1b4-2b2bf9162c13 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:03.327792 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcb0a43-0fca-4bc5-a55d-5b8adeb89d31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:03.328395 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcb0a43-0fca-4bc5-a55d-5b8adeb89d31 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:03.328674 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcb0a43-0fca-4bc5-a55d-5b8adeb89d31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:03.328948 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcb0a43-0fca-4bc5-a55d-5b8adeb89d31 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:03.329574 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dcb0a43-0fca-4bc5-a55d-5b8adeb89d31 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:03.331495 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dcb0a43-0fca-4bc5-a55d-5b8adeb89d31 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:03.363392 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1352/2692] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:26:03 2022] POST /identity/v3/projects => generated 385 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:26:03.365005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ff2e57-b791-4a0c-bbd5-0fbc7468e357 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:03.378641 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ff2e57-b791-4a0c-bbd5-0fbc7468e357 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:03.447849 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c517410-973a-445a-af96-c718a74230b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:03.448601 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c517410-973a-445a-af96-c718a74230b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:03.448982 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c517410-973a-445a-af96-c718a74230b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:03.449953 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c517410-973a-445a-af96-c718a74230b2 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:03.450088 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c517410-973a-445a-af96-c718a74230b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:03.452115 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c517410-973a-445a-af96-c718a74230b2 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:03.481435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1341/2693] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:26:03 2022] POST /identity/v3/users => generated 293 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:26:03.483116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcb0a43-0fca-4bc5-a55d-5b8adeb89d31 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:03.494500 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcb0a43-0fca-4bc5-a55d-5b8adeb89d31 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:03.510881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a961a822-13de-413b-9331-b3a4035e54ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:03.510881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a961a822-13de-413b-9331-b3a4035e54ff admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:03.510881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a961a822-13de-413b-9331-b3a4035e54ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:03.510881 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a961a822-13de-413b-9331-b3a4035e54ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:03.511789 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a961a822-13de-413b-9331-b3a4035e54ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:03.511789 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a961a822-13de-413b-9331-b3a4035e54ff admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:03.523502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1353/2694] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:03 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:26:03.527634 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c517410-973a-445a-af96-c718a74230b2 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:03.539496 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c517410-973a-445a-af96-c718a74230b2 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:03.544431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48dc7a2d-c9d1-4a40-ad01-5bc86d98e282 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:03.545157 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dc7a2d-c9d1-4a40-ad01-5bc86d98e282 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:03.545446 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dc7a2d-c9d1-4a40-ad01-5bc86d98e282 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:03.545877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dc7a2d-c9d1-4a40-ad01-5bc86d98e282 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:03.546399 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48dc7a2d-c9d1-4a40-ad01-5bc86d98e282 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:03.548033 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48dc7a2d-c9d1-4a40-ad01-5bc86d98e282 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:03.560204 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1342/2695] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:03 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:26:03.564498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a961a822-13de-413b-9331-b3a4035e54ff admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:03.574924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a961a822-13de-413b-9331-b3a4035e54ff admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:03.579381 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a0836e-ae2a-45af-8836-852377f6fd08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:03.580014 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a0836e-ae2a-45af-8836-852377f6fd08 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:03.580284 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a0836e-ae2a-45af-8836-852377f6fd08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:03.580551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a0836e-ae2a-45af-8836-852377f6fd08 admin admin] PATH_INFO: `/v3/projects/2c04d8d567d842f5a472031cb9e723a9/users/a2d2b5e6702c4641b6d4e0029a6e9ccb/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:03.595591 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a0836e-ae2a-45af-8836-852377f6fd08 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2c04d8d567d842f5a472031cb9e723a9, user_id=a2d2b5e6702c4641b6d4e0029a6e9ccb, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:03.598242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a0836e-ae2a-45af-8836-852377f6fd08 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:03.610536 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1354/2696] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:26:03 2022] PUT /identity/v3/projects/2c04d8d567d842f5a472031cb9e723a9/users/a2d2b5e6702c4641b6d4e0029a6e9ccb/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:03.620342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8feda500-67b8-4fa7-8b6b-118129bff6f4 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:03.620631 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8feda500-67b8-4fa7-8b6b-118129bff6f4 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:03.620906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8feda500-67b8-4fa7-8b6b-118129bff6f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:03.649333 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8feda500-67b8-4fa7-8b6b-118129bff6f4 None None] MFA Rules not processed for user `a2d2b5e6702c4641b6d4e0029a6e9ccb`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:03.650640 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8feda500-67b8-4fa7-8b6b-118129bff6f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:03.693259 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:03.719439 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1343/2697] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:03 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:03.723894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a0836e-ae2a-45af-8836-852377f6fd08 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:03.801397 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a0836e-ae2a-45af-8836-852377f6fd08 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:03.806647 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1651cf69-5d6d-4828-a4f4-d77ba7506521 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:03.807417 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1651cf69-5d6d-4828-a4f4-d77ba7506521 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:03.807605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1651cf69-5d6d-4828-a4f4-d77ba7506521 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:03.807886 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1651cf69-5d6d-4828-a4f4-d77ba7506521 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:03.808503 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1651cf69-5d6d-4828-a4f4-d77ba7506521 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:03.823131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1651cf69-5d6d-4828-a4f4-d77ba7506521 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:03.823131 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:03.840565 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1355/2698] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:26:03 2022] POST /identity/v3/projects => generated 383 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:26:03.844528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8feda500-67b8-4fa7-8b6b-118129bff6f4 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:03.856222 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8feda500-67b8-4fa7-8b6b-118129bff6f4 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:03.915962 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f1b57b-f9c3-44c8-91e4-81e48a74c7ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:03.916842 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f1b57b-f9c3-44c8-91e4-81e48a74c7ed admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:03.917122 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f1b57b-f9c3-44c8-91e4-81e48a74c7ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:03.917393 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f1b57b-f9c3-44c8-91e4-81e48a74c7ed admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:03.918027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f1b57b-f9c3-44c8-91e4-81e48a74c7ed admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:03.919940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f1b57b-f9c3-44c8-91e4-81e48a74c7ed admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:03.948178 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1344/2699] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:26:03 2022] POST /identity/v3/users => generated 292 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:26:03.949483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1651cf69-5d6d-4828-a4f4-d77ba7506521 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:03.961115 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1651cf69-5d6d-4828-a4f4-d77ba7506521 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:03.966179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899ce733-a276-4d9b-acaa-e3c21b3d9301 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:03.967136 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899ce733-a276-4d9b-acaa-e3c21b3d9301 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:03.967486 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899ce733-a276-4d9b-acaa-e3c21b3d9301 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:03.967777 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899ce733-a276-4d9b-acaa-e3c21b3d9301 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:03.968424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899ce733-a276-4d9b-acaa-e3c21b3d9301 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:03.970036 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899ce733-a276-4d9b-acaa-e3c21b3d9301 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:03.982884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1356/2700] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:26:03.985395 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f1b57b-f9c3-44c8-91e4-81e48a74c7ed admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:03.997055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f1b57b-f9c3-44c8-91e4-81e48a74c7ed admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:04.003010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007556aa-28ff-4dfc-a536-81dfba7d0cee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:04.003620 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007556aa-28ff-4dfc-a536-81dfba7d0cee admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:04.003907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007556aa-28ff-4dfc-a536-81dfba7d0cee admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:04.004177 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007556aa-28ff-4dfc-a536-81dfba7d0cee admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:04.004841 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007556aa-28ff-4dfc-a536-81dfba7d0cee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:04.006445 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007556aa-28ff-4dfc-a536-81dfba7d0cee admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:04.017220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1345/2701] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:26:04.022294 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899ce733-a276-4d9b-acaa-e3c21b3d9301 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:04.032787 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899ce733-a276-4d9b-acaa-e3c21b3d9301 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:04.038170 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d48cda-be01-4c64-b350-7c32b408952a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:04.038901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d48cda-be01-4c64-b350-7c32b408952a admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:04.039572 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d48cda-be01-4c64-b350-7c32b408952a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:04.039773 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d48cda-be01-4c64-b350-7c32b408952a admin admin] PATH_INFO: `/v3/projects/051aa6fe9fac451f9c5621b1ac88a602/users/f62cda1c1caf4cba9352596e3b443550/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:04.056393 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d48cda-be01-4c64-b350-7c32b408952a admin admin] RBAC: Authorizing `identity:create_grant(project_id=051aa6fe9fac451f9c5621b1ac88a602, user_id=f62cda1c1caf4cba9352596e3b443550, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:04.059280 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d48cda-be01-4c64-b350-7c32b408952a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:04.072267 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1357/2702] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:26:04 2022] PUT /identity/v3/projects/051aa6fe9fac451f9c5621b1ac88a602/users/f62cda1c1caf4cba9352596e3b443550/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:04.083610 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f36288b-1ca9-41ae-bee0-40efba5dda21 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:04.083833 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f36288b-1ca9-41ae-bee0-40efba5dda21 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:04.084103 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f36288b-1ca9-41ae-bee0-40efba5dda21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:04.112167 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9f36288b-1ca9-41ae-bee0-40efba5dda21 None None] MFA Rules not processed for user `f62cda1c1caf4cba9352596e3b443550`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:04.113381 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9f36288b-1ca9-41ae-bee0-40efba5dda21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:04.178988 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1346/2703] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:04 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:04.183841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d48cda-be01-4c64-b350-7c32b408952a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:04.244922 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d48cda-be01-4c64-b350-7c32b408952a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:04.249664 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abb0431-29ad-4a60-a339-99b73d571ac7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:04.250301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abb0431-29ad-4a60-a339-99b73d571ac7 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:04.250582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abb0431-29ad-4a60-a339-99b73d571ac7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:04.250894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abb0431-29ad-4a60-a339-99b73d571ac7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:04.251525 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abb0431-29ad-4a60-a339-99b73d571ac7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:04.253539 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abb0431-29ad-4a60-a339-99b73d571ac7 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:04.281213 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1358/2704] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:26:04 2022] POST /identity/v3/projects => generated 383 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:26:04.285216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f36288b-1ca9-41ae-bee0-40efba5dda21 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:04.295724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f36288b-1ca9-41ae-bee0-40efba5dda21 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:04.339582 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:04.351565 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464b9b09-35b3-49ce-acb1-d667196a4231 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:04.352188 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464b9b09-35b3-49ce-acb1-d667196a4231 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:04.352463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464b9b09-35b3-49ce-acb1-d667196a4231 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:04.352730 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464b9b09-35b3-49ce-acb1-d667196a4231 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:04.353339 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464b9b09-35b3-49ce-acb1-d667196a4231 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:04.355274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464b9b09-35b3-49ce-acb1-d667196a4231 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:04.375354 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1347/2705] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:26:04 2022] POST /identity/v3/users => generated 298 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:26:04.379407 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abb0431-29ad-4a60-a339-99b73d571ac7 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:04.390149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abb0431-29ad-4a60-a339-99b73d571ac7 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:04.395000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587f0137-332a-43c0-be8a-3f7ef32277d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:04.396269 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587f0137-332a-43c0-be8a-3f7ef32277d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:04.396546 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587f0137-332a-43c0-be8a-3f7ef32277d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:04.396816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587f0137-332a-43c0-be8a-3f7ef32277d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:04.397477 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587f0137-332a-43c0-be8a-3f7ef32277d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:04.399105 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587f0137-332a-43c0-be8a-3f7ef32277d3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:04.410353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1359/2706] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:04 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:26:04.414268 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464b9b09-35b3-49ce-acb1-d667196a4231 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:04.424398 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464b9b09-35b3-49ce-acb1-d667196a4231 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:04.429434 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7626516-5bbd-4939-87a9-0acc4a78c872 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:04.430340 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7626516-5bbd-4939-87a9-0acc4a78c872 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:04.430617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7626516-5bbd-4939-87a9-0acc4a78c872 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:04.430915 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7626516-5bbd-4939-87a9-0acc4a78c872 admin admin] PATH_INFO: `/v3/projects/04c0f61894b04619b9e9486dd55b3bbd/users/d5d28a7157b64118b3529a2f5d70af94/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:04.445980 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7626516-5bbd-4939-87a9-0acc4a78c872 admin admin] RBAC: Authorizing `identity:create_grant(project_id=04c0f61894b04619b9e9486dd55b3bbd, user_id=d5d28a7157b64118b3529a2f5d70af94, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:04.448635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7626516-5bbd-4939-87a9-0acc4a78c872 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:04.461055 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1348/2707] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:26:04 2022] PUT /identity/v3/projects/04c0f61894b04619b9e9486dd55b3bbd/users/d5d28a7157b64118b3529a2f5d70af94/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:04.464874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587f0137-332a-43c0-be8a-3f7ef32277d3 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:04.474660 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587f0137-332a-43c0-be8a-3f7ef32277d3 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:04.531433 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03dff45-0167-44c3-ad68-4552606ac043 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:04.532034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03dff45-0167-44c3-ad68-4552606ac043 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:04.532308 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03dff45-0167-44c3-ad68-4552606ac043 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:04.532597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03dff45-0167-44c3-ad68-4552606ac043 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:04.533225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03dff45-0167-44c3-ad68-4552606ac043 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:04.534794 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03dff45-0167-44c3-ad68-4552606ac043 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:04.545311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1360/2708] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:04 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:26:04.549532 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7626516-5bbd-4939-87a9-0acc4a78c872 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:04.561095 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7626516-5bbd-4939-87a9-0acc4a78c872 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:04.565596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be180342-6025-417c-83aa-fd8cd6ccde84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:04.566233 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be180342-6025-417c-83aa-fd8cd6ccde84 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:04.566509 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be180342-6025-417c-83aa-fd8cd6ccde84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:04.566777 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be180342-6025-417c-83aa-fd8cd6ccde84 admin admin] PATH_INFO: `/v3/projects/04c0f61894b04619b9e9486dd55b3bbd/users/d5d28a7157b64118b3529a2f5d70af94/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:04.569313 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be180342-6025-417c-83aa-fd8cd6ccde84 admin admin] RBAC: Authorizing `identity:create_grant(project_id=04c0f61894b04619b9e9486dd55b3bbd, user_id=d5d28a7157b64118b3529a2f5d70af94, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:04.571996 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be180342-6025-417c-83aa-fd8cd6ccde84 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:04.582669 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1349/2709] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:26:04 2022] PUT /identity/v3/projects/04c0f61894b04619b9e9486dd55b3bbd/users/d5d28a7157b64118b3529a2f5d70af94/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:04.594059 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4214b87f-c773-4c85-9624-64ffc08f4ef7 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:04.594274 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4214b87f-c773-4c85-9624-64ffc08f4ef7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:04.594545 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4214b87f-c773-4c85-9624-64ffc08f4ef7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:04.623045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4214b87f-c773-4c85-9624-64ffc08f4ef7 None None] MFA Rules not processed for user `d5d28a7157b64118b3529a2f5d70af94`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:04.624257 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4214b87f-c773-4c85-9624-64ffc08f4ef7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:04.634183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:04.701037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1361/2710] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:04 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:26:04.711891 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be180342-6025-417c-83aa-fd8cd6ccde84 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:04.713662 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-be180342-6025-417c-83aa-fd8cd6ccde84 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:04.774225 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be180342-6025-417c-83aa-fd8cd6ccde84 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:04.777860 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8458bc1e-c167-429f-88ef-549d14f96faa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:04.778609 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8458bc1e-c167-429f-88ef-549d14f96faa service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:04.778909 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8458bc1e-c167-429f-88ef-549d14f96faa service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:04.779488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8458bc1e-c167-429f-88ef-549d14f96faa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:04.844173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8458bc1e-c167-429f-88ef-549d14f96faa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:04.845541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8458bc1e-c167-429f-88ef-549d14f96faa service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:04.856024 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1350/2711] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:04 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:26:08.490878 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4214b87f-c773-4c85-9624-64ffc08f4ef7 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:08.501363 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4214b87f-c773-4c85-9624-64ffc08f4ef7 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:08.507779 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9341ca66-8161-49c3-acb4-376f1330b014 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:08.508421 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9341ca66-8161-49c3-acb4-376f1330b014 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:08.508697 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9341ca66-8161-49c3-acb4-376f1330b014 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:08.509104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9341ca66-8161-49c3-acb4-376f1330b014 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:08.574754 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9341ca66-8161-49c3-acb4-376f1330b014 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:08.576407 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9341ca66-8161-49c3-acb4-376f1330b014 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:08.600981 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1362/2712] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:08 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:26:09.152654 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8458bc1e-c167-429f-88ef-549d14f96faa service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:09.161080 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8458bc1e-c167-429f-88ef-549d14f96faa service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:09.165538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb61c5a-76c6-4149-9273-4c9f15d00283 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:09.166166 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb61c5a-76c6-4149-9273-4c9f15d00283 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:09.166487 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb61c5a-76c6-4149-9273-4c9f15d00283 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:09.166779 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb61c5a-76c6-4149-9273-4c9f15d00283 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:09.176845 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb61c5a-76c6-4149-9273-4c9f15d00283 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:09.178743 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb61c5a-76c6-4149-9273-4c9f15d00283 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:09.188292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1351/2713] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:09 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:26:13.151597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9341ca66-8161-49c3-acb4-376f1330b014 service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:13.230311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9341ca66-8161-49c3-acb4-376f1330b014 service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:13.235509 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a076b0-1645-46f0-a7fb-4a24523a9ce1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:13.236311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a076b0-1645-46f0-a7fb-4a24523a9ce1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:13.236610 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a076b0-1645-46f0-a7fb-4a24523a9ce1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:13.236883 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a076b0-1645-46f0-a7fb-4a24523a9ce1 admin admin] PATH_INFO: `/v3/users/a2d2b5e6702c4641b6d4e0029a6e9ccb` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:13.238061 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a076b0-1645-46f0-a7fb-4a24523a9ce1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a2d2b5e6702c4641b6d4e0029a6e9ccb)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:13.240631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a076b0-1645-46f0-a7fb-4a24523a9ce1 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:13.296595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d4a076b0-1645-46f0-a7fb-4a24523a9ce1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a2d2b5e6702c4641b6d4e0029a6e9ccb'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:13.297329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d4a076b0-1645-46f0-a7fb-4a24523a9ce1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a2d2b5e6702c4641b6d4e0029a6e9ccb'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:13.299581 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:13.309157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d4a076b0-1645-46f0-a7fb-4a24523a9ce1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a2d2b5e6702c4641b6d4e0029a6e9ccb'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:13.310527 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d4a076b0-1645-46f0-a7fb-4a24523a9ce1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a2d2b5e6702c4641b6d4e0029a6e9ccb'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:13.326769 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-d4a076b0-1645-46f0-a7fb-4a24523a9ce1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a2d2b5e6702c4641b6d4e0029a6e9ccb'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:13.340786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1363/2714] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:26:13 2022] DELETE /identity/v3/users/a2d2b5e6702c4641b6d4e0029a6e9ccb => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:13.345646 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee85342c-1af4-4663-876c-f18662a352f2 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:13.346175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee85342c-1af4-4663-876c-f18662a352f2 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:13.346284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee85342c-1af4-4663-876c-f18662a352f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:13.368431 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-ee85342c-1af4-4663-876c-f18662a352f2 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:13.369815 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ee85342c-1af4-4663-876c-f18662a352f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:13.429355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:13.463126 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1352/2715] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:13.473185 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a076b0-1645-46f0-a7fb-4a24523a9ce1 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:13.475356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d4a076b0-1645-46f0-a7fb-4a24523a9ce1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:13.547800 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a076b0-1645-46f0-a7fb-4a24523a9ce1 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:13.551689 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7831e2-1416-46ba-925e-096998d840d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:13.552391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7831e2-1416-46ba-925e-096998d840d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:13.552664 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7831e2-1416-46ba-925e-096998d840d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:13.552928 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7831e2-1416-46ba-925e-096998d840d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:13.562168 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7831e2-1416-46ba-925e-096998d840d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:13.563732 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7831e2-1416-46ba-925e-096998d840d2 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:13.573836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1364/2716] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:26:13.847463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee85342c-1af4-4663-876c-f18662a352f2 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:13.849447 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ee85342c-1af4-4663-876c-f18662a352f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:13.861435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee85342c-1af4-4663-876c-f18662a352f2 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:13.866795 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b25baed-99bb-4de8-92d5-3c770d93fc85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:13.867458 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b25baed-99bb-4de8-92d5-3c770d93fc85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:13.867737 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b25baed-99bb-4de8-92d5-3c770d93fc85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:13.868021 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b25baed-99bb-4de8-92d5-3c770d93fc85 admin admin] PATH_INFO: `/v3/projects/2c04d8d567d842f5a472031cb9e723a9` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:13.869122 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b25baed-99bb-4de8-92d5-3c770d93fc85 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2c04d8d567d842f5a472031cb9e723a9)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:13.871065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b25baed-99bb-4de8-92d5-3c770d93fc85 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:13.926462 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7b25baed-99bb-4de8-92d5-3c770d93fc85 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2c04d8d567d842f5a472031cb9e723a9', 'request_id': 'req-7b25baed-99bb-4de8-92d5-3c770d93fc85'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:13.933279 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7b25baed-99bb-4de8-92d5-3c770d93fc85 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2c04d8d567d842f5a472031cb9e723a9', 'request_id': 'req-7b25baed-99bb-4de8-92d5-3c770d93fc85'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:13.940568 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7b25baed-99bb-4de8-92d5-3c770d93fc85 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2c04d8d567d842f5a472031cb9e723a9', 'request_id': 'req-7b25baed-99bb-4de8-92d5-3c770d93fc85'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:13.955965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7b25baed-99bb-4de8-92d5-3c770d93fc85 admin admin] The token cache is being invalidate because project 2c04d8d567d842f5a472031cb9e723a9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:26:13.956336 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7b25baed-99bb-4de8-92d5-3c770d93fc85 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:13.957110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7b25baed-99bb-4de8-92d5-3c770d93fc85 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:13.958715 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1353/2717] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:26:13 2022] DELETE /identity/v3/projects/2c04d8d567d842f5a472031cb9e723a9 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:13.962529 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7831e2-1416-46ba-925e-096998d840d2 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:13.972129 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7831e2-1416-46ba-925e-096998d840d2 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:14.032044 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df09973b-85f3-47ad-b006-8596a5f6a4e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:14.032678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df09973b-85f3-47ad-b006-8596a5f6a4e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:14.032953 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df09973b-85f3-47ad-b006-8596a5f6a4e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:14.033351 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df09973b-85f3-47ad-b006-8596a5f6a4e9 admin admin] PATH_INFO: `/v3/users/f62cda1c1caf4cba9352596e3b443550` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:14.034491 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df09973b-85f3-47ad-b006-8596a5f6a4e9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f62cda1c1caf4cba9352596e3b443550)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:14.036978 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df09973b-85f3-47ad-b006-8596a5f6a4e9 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:14.104941 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-df09973b-85f3-47ad-b006-8596a5f6a4e9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f62cda1c1caf4cba9352596e3b443550'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:14.105658 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-df09973b-85f3-47ad-b006-8596a5f6a4e9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f62cda1c1caf4cba9352596e3b443550'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:14.113707 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-df09973b-85f3-47ad-b006-8596a5f6a4e9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f62cda1c1caf4cba9352596e3b443550'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:14.114386 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-df09973b-85f3-47ad-b006-8596a5f6a4e9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f62cda1c1caf4cba9352596e3b443550'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:14.129627 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:INSERT=2,keystone:DELETE=13,keystone:SELECT=85 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:14.131033 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-df09973b-85f3-47ad-b006-8596a5f6a4e9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f62cda1c1caf4cba9352596e3b443550'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:14.146398 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1365/2718] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:26:13 2022] DELETE /identity/v3/users/f62cda1c1caf4cba9352596e3b443550 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:14.409196 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b25baed-99bb-4de8-92d5-3c770d93fc85 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:14.410874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-7b25baed-99bb-4de8-92d5-3c770d93fc85 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:14.426674 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b25baed-99bb-4de8-92d5-3c770d93fc85 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:14.488162 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0606a421-6b7e-4f5c-846c-8bc885a5897a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:14.488897 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0606a421-6b7e-4f5c-846c-8bc885a5897a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:14.489262 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0606a421-6b7e-4f5c-846c-8bc885a5897a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:14.489557 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0606a421-6b7e-4f5c-846c-8bc885a5897a admin admin] PATH_INFO: `/v3/projects/051aa6fe9fac451f9c5621b1ac88a602` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:14.490663 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0606a421-6b7e-4f5c-846c-8bc885a5897a admin admin] RBAC: Authorizing `identity:delete_project(project_id=051aa6fe9fac451f9c5621b1ac88a602)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:14.492735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0606a421-6b7e-4f5c-846c-8bc885a5897a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:14.542025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0606a421-6b7e-4f5c-846c-8bc885a5897a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '051aa6fe9fac451f9c5621b1ac88a602', 'request_id': 'req-0606a421-6b7e-4f5c-846c-8bc885a5897a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:14.550592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0606a421-6b7e-4f5c-846c-8bc885a5897a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '051aa6fe9fac451f9c5621b1ac88a602', 'request_id': 'req-0606a421-6b7e-4f5c-846c-8bc885a5897a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:14.568317 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0606a421-6b7e-4f5c-846c-8bc885a5897a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '051aa6fe9fac451f9c5621b1ac88a602', 'request_id': 'req-0606a421-6b7e-4f5c-846c-8bc885a5897a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:14.568317 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:14.573740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0606a421-6b7e-4f5c-846c-8bc885a5897a admin admin] The token cache is being invalidate because project 051aa6fe9fac451f9c5621b1ac88a602 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:26:14.574533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0606a421-6b7e-4f5c-846c-8bc885a5897a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:14.575419 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-0606a421-6b7e-4f5c-846c-8bc885a5897a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:14.577099 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1354/2719] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:26:14 2022] DELETE /identity/v3/projects/051aa6fe9fac451f9c5621b1ac88a602 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:14.581119 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df09973b-85f3-47ad-b006-8596a5f6a4e9 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:14.583132 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-df09973b-85f3-47ad-b006-8596a5f6a4e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:14.617114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df09973b-85f3-47ad-b006-8596a5f6a4e9 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:14.689400 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd6f408-aa80-47ce-be5e-0d7fb21e845e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:14.690155 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd6f408-aa80-47ce-be5e-0d7fb21e845e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:14.690536 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd6f408-aa80-47ce-be5e-0d7fb21e845e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:14.690941 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd6f408-aa80-47ce-be5e-0d7fb21e845e admin admin] PATH_INFO: `/v3/users/d5d28a7157b64118b3529a2f5d70af94` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:14.692164 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd6f408-aa80-47ce-be5e-0d7fb21e845e admin admin] RBAC: Authorizing `identity:delete_user(user_id=d5d28a7157b64118b3529a2f5d70af94)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:14.694793 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd6f408-aa80-47ce-be5e-0d7fb21e845e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:14.749946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-acd6f408-aa80-47ce-be5e-0d7fb21e845e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd5d28a7157b64118b3529a2f5d70af94'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:14.750894 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-acd6f408-aa80-47ce-be5e-0d7fb21e845e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd5d28a7157b64118b3529a2f5d70af94'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:14.761411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-acd6f408-aa80-47ce-be5e-0d7fb21e845e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd5d28a7157b64118b3529a2f5d70af94'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:14.762808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-acd6f408-aa80-47ce-be5e-0d7fb21e845e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd5d28a7157b64118b3529a2f5d70af94'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:14.778179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-acd6f408-aa80-47ce-be5e-0d7fb21e845e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd5d28a7157b64118b3529a2f5d70af94'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:14.791406 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1366/2720] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:26:14 2022] DELETE /identity/v3/users/d5d28a7157b64118b3529a2f5d70af94 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:15.366316 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0606a421-6b7e-4f5c-846c-8bc885a5897a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:15.367823 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0606a421-6b7e-4f5c-846c-8bc885a5897a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:15.386712 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0606a421-6b7e-4f5c-846c-8bc885a5897a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:15.464371 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cde6445-dddf-471f-9967-de8cabfb80ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:15.466073 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cde6445-dddf-471f-9967-de8cabfb80ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:15.467193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cde6445-dddf-471f-9967-de8cabfb80ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:15.467911 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cde6445-dddf-471f-9967-de8cabfb80ef admin admin] PATH_INFO: `/v3/projects/04c0f61894b04619b9e9486dd55b3bbd` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:15.469310 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cde6445-dddf-471f-9967-de8cabfb80ef admin admin] RBAC: Authorizing `identity:delete_project(project_id=04c0f61894b04619b9e9486dd55b3bbd)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:15.471480 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cde6445-dddf-471f-9967-de8cabfb80ef admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:15.538182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5cde6445-dddf-471f-9967-de8cabfb80ef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '04c0f61894b04619b9e9486dd55b3bbd', 'request_id': 'req-5cde6445-dddf-471f-9967-de8cabfb80ef'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:15.542207 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5cde6445-dddf-471f-9967-de8cabfb80ef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '04c0f61894b04619b9e9486dd55b3bbd', 'request_id': 'req-5cde6445-dddf-471f-9967-de8cabfb80ef'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:15.550254 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5cde6445-dddf-471f-9967-de8cabfb80ef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '04c0f61894b04619b9e9486dd55b3bbd', 'request_id': 'req-5cde6445-dddf-471f-9967-de8cabfb80ef'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:15.573259 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5cde6445-dddf-471f-9967-de8cabfb80ef admin admin] The token cache is being invalidate because project 04c0f61894b04619b9e9486dd55b3bbd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:26:15.573551 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5cde6445-dddf-471f-9967-de8cabfb80ef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:15.574215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-5cde6445-dddf-471f-9967-de8cabfb80ef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:15.575981 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1355/2721] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:26:15 2022] DELETE /identity/v3/projects/04c0f61894b04619b9e9486dd55b3bbd => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:15.580613 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b6658f-60db-428f-b5fc-c788e523f1d8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:15.580966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b6658f-60db-428f-b5fc-c788e523f1d8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:15.581800 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b6658f-60db-428f-b5fc-c788e523f1d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:15.614008 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-a0b6658f-60db-428f-b5fc-c788e523f1d8 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:15.615370 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a0b6658f-60db-428f-b5fc-c788e523f1d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:15.695169 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1367/2722] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:15.703145 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689fd437-0055-46f0-b088-3599abe30058 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:15.703469 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689fd437-0055-46f0-b088-3599abe30058 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:15.703752 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689fd437-0055-46f0-b088-3599abe30058 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:15.724127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-689fd437-0055-46f0-b088-3599abe30058 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:15.725341 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-689fd437-0055-46f0-b088-3599abe30058 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:15.733526 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1356/2723] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:15.737747 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b6658f-60db-428f-b5fc-c788e523f1d8 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:15.748378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b6658f-60db-428f-b5fc-c788e523f1d8 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:15.753665 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940a45d9-3f8e-4e97-be81-3392eb6a0084 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:15.754266 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940a45d9-3f8e-4e97-be81-3392eb6a0084 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:15.754544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940a45d9-3f8e-4e97-be81-3392eb6a0084 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:15.754814 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940a45d9-3f8e-4e97-be81-3392eb6a0084 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:15.756116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940a45d9-3f8e-4e97-be81-3392eb6a0084 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:26:15.756416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940a45d9-3f8e-4e97-be81-3392eb6a0084 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:15.757980 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940a45d9-3f8e-4e97-be81-3392eb6a0084 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:15.759614 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:15.775028 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1368/2724] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:26:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:26:15.776455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689fd437-0055-46f0-b088-3599abe30058 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:15.785819 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689fd437-0055-46f0-b088-3599abe30058 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:15.790907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8195d47-3d51-4ac9-86f3-c59b038c5505 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:15.791517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8195d47-3d51-4ac9-86f3-c59b038c5505 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:15.791796 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8195d47-3d51-4ac9-86f3-c59b038c5505 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:15.792070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8195d47-3d51-4ac9-86f3-c59b038c5505 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:15.792735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8195d47-3d51-4ac9-86f3-c59b038c5505 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:15.794877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8195d47-3d51-4ac9-86f3-c59b038c5505 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:15.829651 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1357/2725] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:26:15 2022] POST /identity/v3/projects => generated 391 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:26:15.834541 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940a45d9-3f8e-4e97-be81-3392eb6a0084 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:15.845202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940a45d9-3f8e-4e97-be81-3392eb6a0084 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:15.905864 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a404539-7706-4046-a59c-f6ac891eab4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:15.906569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a404539-7706-4046-a59c-f6ac891eab4f admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:15.906884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a404539-7706-4046-a59c-f6ac891eab4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:15.907148 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a404539-7706-4046-a59c-f6ac891eab4f admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:15.907793 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a404539-7706-4046-a59c-f6ac891eab4f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:15.909723 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a404539-7706-4046-a59c-f6ac891eab4f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:15.932771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1369/2726] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:26:15 2022] POST /identity/v3/users => generated 296 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:26:15.937317 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8195d47-3d51-4ac9-86f3-c59b038c5505 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:15.948856 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8195d47-3d51-4ac9-86f3-c59b038c5505 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:15.953540 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2df8e83-cdba-4ce4-a07e-7df3f7c0e8e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:15.954166 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2df8e83-cdba-4ce4-a07e-7df3f7c0e8e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:15.954426 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2df8e83-cdba-4ce4-a07e-7df3f7c0e8e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:15.954695 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2df8e83-cdba-4ce4-a07e-7df3f7c0e8e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:15.955356 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2df8e83-cdba-4ce4-a07e-7df3f7c0e8e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:15.956982 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2df8e83-cdba-4ce4-a07e-7df3f7c0e8e9 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:15.970763 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1358/2727] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:15 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:26:15.972126 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a404539-7706-4046-a59c-f6ac891eab4f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:15.982655 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a404539-7706-4046-a59c-f6ac891eab4f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:15.987346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8794c0e6-1939-45c7-8616-88ba1dd2d3b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:15.987963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8794c0e6-1939-45c7-8616-88ba1dd2d3b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:15.988242 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8794c0e6-1939-45c7-8616-88ba1dd2d3b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:15.988515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8794c0e6-1939-45c7-8616-88ba1dd2d3b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:15.989191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8794c0e6-1939-45c7-8616-88ba1dd2d3b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:15.991313 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8794c0e6-1939-45c7-8616-88ba1dd2d3b1 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:16.002743 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1370/2728] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:15 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:26:16.007966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2df8e83-cdba-4ce4-a07e-7df3f7c0e8e9 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:16.018194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2df8e83-cdba-4ce4-a07e-7df3f7c0e8e9 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:16.022773 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106046da-e840-46ef-88b7-d7f077aac186 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:16.023456 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106046da-e840-46ef-88b7-d7f077aac186 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:16.023733 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106046da-e840-46ef-88b7-d7f077aac186 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:16.024003 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106046da-e840-46ef-88b7-d7f077aac186 admin admin] PATH_INFO: `/v3/projects/800746745cc74a04ae43f97a17f1a722/users/da3fd9b1ea02427fb371cf19affd12b8/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:16.039923 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106046da-e840-46ef-88b7-d7f077aac186 admin admin] RBAC: Authorizing `identity:create_grant(project_id=800746745cc74a04ae43f97a17f1a722, user_id=da3fd9b1ea02427fb371cf19affd12b8, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:16.042577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106046da-e840-46ef-88b7-d7f077aac186 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:16.057077 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1359/2729] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:26:16 2022] PUT /identity/v3/projects/800746745cc74a04ae43f97a17f1a722/users/da3fd9b1ea02427fb371cf19affd12b8/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:16.068537 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d12ee2-eefa-4745-a819-7c769a05b59c None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:16.068898 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d12ee2-eefa-4745-a819-7c769a05b59c None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:16.069264 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d12ee2-eefa-4745-a819-7c769a05b59c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:16.096774 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-89d12ee2-eefa-4745-a819-7c769a05b59c None None] MFA Rules not processed for user `da3fd9b1ea02427fb371cf19affd12b8`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:16.098015 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-89d12ee2-eefa-4745-a819-7c769a05b59c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:16.165399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1371/2730] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:16 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:16.176745 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106046da-e840-46ef-88b7-d7f077aac186 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:16.178540 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-106046da-e840-46ef-88b7-d7f077aac186 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:16.204925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:16.239040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106046da-e840-46ef-88b7-d7f077aac186 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:16.242748 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97b3d45-71c0-4e7c-befc-ed84a7e23462 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:16.243383 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97b3d45-71c0-4e7c-befc-ed84a7e23462 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:16.243656 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97b3d45-71c0-4e7c-befc-ed84a7e23462 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:16.243925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97b3d45-71c0-4e7c-befc-ed84a7e23462 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:16.252780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97b3d45-71c0-4e7c-befc-ed84a7e23462 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:16.254240 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97b3d45-71c0-4e7c-befc-ed84a7e23462 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:16.265218 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1360/2731] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:16 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:26:17.647198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d12ee2-eefa-4745-a819-7c769a05b59c None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:17.648822 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-89d12ee2-eefa-4745-a819-7c769a05b59c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:17.732010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d12ee2-eefa-4745-a819-7c769a05b59c None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:17.736565 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f55ce03-b58a-4ef1-b28b-0bedb4f8a813 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:17.737192 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f55ce03-b58a-4ef1-b28b-0bedb4f8a813 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:17.737465 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f55ce03-b58a-4ef1-b28b-0bedb4f8a813 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:17.737731 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f55ce03-b58a-4ef1-b28b-0bedb4f8a813 admin admin] PATH_INFO: `/v3/users/b64c3c41193f44aca60d1622afaa0175` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:17.738838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f55ce03-b58a-4ef1-b28b-0bedb4f8a813 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b64c3c41193f44aca60d1622afaa0175)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:17.741342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f55ce03-b58a-4ef1-b28b-0bedb4f8a813 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:17.744109 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:17.799947 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2f55ce03-b58a-4ef1-b28b-0bedb4f8a813 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b64c3c41193f44aca60d1622afaa0175'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:17.800735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2f55ce03-b58a-4ef1-b28b-0bedb4f8a813 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b64c3c41193f44aca60d1622afaa0175'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:17.809759 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2f55ce03-b58a-4ef1-b28b-0bedb4f8a813 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b64c3c41193f44aca60d1622afaa0175'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:17.810477 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2f55ce03-b58a-4ef1-b28b-0bedb4f8a813 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b64c3c41193f44aca60d1622afaa0175'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:17.831718 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-2f55ce03-b58a-4ef1-b28b-0bedb4f8a813 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b64c3c41193f44aca60d1622afaa0175'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:17.849134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1372/2732] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:26:17 2022] DELETE /identity/v3/users/b64c3c41193f44aca60d1622afaa0175 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:26:18.044072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97b3d45-71c0-4e7c-befc-ed84a7e23462 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:18.055784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97b3d45-71c0-4e7c-befc-ed84a7e23462 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:18.128348 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5070ce2-2bf3-476a-9a21-c64bc7af0540 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:18.129007 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5070ce2-2bf3-476a-9a21-c64bc7af0540 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:18.129284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5070ce2-2bf3-476a-9a21-c64bc7af0540 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:18.129556 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5070ce2-2bf3-476a-9a21-c64bc7af0540 admin admin] PATH_INFO: `/v3/projects/5b9f4c16910f4484af30b5757bbc27f2` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:18.130686 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5070ce2-2bf3-476a-9a21-c64bc7af0540 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5b9f4c16910f4484af30b5757bbc27f2)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:18.132613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5070ce2-2bf3-476a-9a21-c64bc7af0540 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:18.184359 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a5070ce2-2bf3-476a-9a21-c64bc7af0540 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5b9f4c16910f4484af30b5757bbc27f2', 'request_id': 'req-a5070ce2-2bf3-476a-9a21-c64bc7af0540'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:18.189927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a5070ce2-2bf3-476a-9a21-c64bc7af0540 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5b9f4c16910f4484af30b5757bbc27f2', 'request_id': 'req-a5070ce2-2bf3-476a-9a21-c64bc7af0540'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:18.197496 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a5070ce2-2bf3-476a-9a21-c64bc7af0540 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5b9f4c16910f4484af30b5757bbc27f2', 'request_id': 'req-a5070ce2-2bf3-476a-9a21-c64bc7af0540'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:18.211049 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a5070ce2-2bf3-476a-9a21-c64bc7af0540 admin admin] The token cache is being invalidate because project 5b9f4c16910f4484af30b5757bbc27f2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:26:18.211341 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a5070ce2-2bf3-476a-9a21-c64bc7af0540 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:18.211971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a5070ce2-2bf3-476a-9a21-c64bc7af0540 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:18.213449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1361/2733] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:26:18 2022] DELETE /identity/v3/projects/5b9f4c16910f4484af30b5757bbc27f2 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:18.217540 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f55ce03-b58a-4ef1-b28b-0bedb4f8a813 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:18.219054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2f55ce03-b58a-4ef1-b28b-0bedb4f8a813 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:18.234717 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f55ce03-b58a-4ef1-b28b-0bedb4f8a813 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:18.297420 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c59807-c63f-41d2-8741-0406754b692f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:18.298055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c59807-c63f-41d2-8741-0406754b692f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:18.298332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c59807-c63f-41d2-8741-0406754b692f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:18.298601 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c59807-c63f-41d2-8741-0406754b692f admin admin] PATH_INFO: `/v3/users/3412de595c01427486f5fc51bb2cf283` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:18.299762 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c59807-c63f-41d2-8741-0406754b692f admin admin] RBAC: Authorizing `identity:delete_user(user_id=3412de595c01427486f5fc51bb2cf283)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:18.301675 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c59807-c63f-41d2-8741-0406754b692f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:18.351855 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b4c59807-c63f-41d2-8741-0406754b692f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3412de595c01427486f5fc51bb2cf283'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:18.352502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b4c59807-c63f-41d2-8741-0406754b692f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3412de595c01427486f5fc51bb2cf283'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:18.360000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b4c59807-c63f-41d2-8741-0406754b692f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3412de595c01427486f5fc51bb2cf283'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:18.360616 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b4c59807-c63f-41d2-8741-0406754b692f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3412de595c01427486f5fc51bb2cf283'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:18.372833 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b4c59807-c63f-41d2-8741-0406754b692f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3412de595c01427486f5fc51bb2cf283'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:18.381585 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:18.386207 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1373/2734] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:26:18 2022] DELETE /identity/v3/users/3412de595c01427486f5fc51bb2cf283 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:18.567281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5070ce2-2bf3-476a-9a21-c64bc7af0540 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:18.569254 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a5070ce2-2bf3-476a-9a21-c64bc7af0540 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:18.582349 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5070ce2-2bf3-476a-9a21-c64bc7af0540 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:18.634468 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:18.643139 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef1fdb1-4904-4c2f-92cf-9ee6ea263181 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:18.643977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef1fdb1-4904-4c2f-92cf-9ee6ea263181 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:18.644463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef1fdb1-4904-4c2f-92cf-9ee6ea263181 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:18.644893 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef1fdb1-4904-4c2f-92cf-9ee6ea263181 admin admin] PATH_INFO: `/v3/projects/6416e876babe4a30b21eaaa002360013` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:18.646613 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef1fdb1-4904-4c2f-92cf-9ee6ea263181 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6416e876babe4a30b21eaaa002360013)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:18.648594 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef1fdb1-4904-4c2f-92cf-9ee6ea263181 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:18.694412 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4ef1fdb1-4904-4c2f-92cf-9ee6ea263181 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6416e876babe4a30b21eaaa002360013', 'request_id': 'req-4ef1fdb1-4904-4c2f-92cf-9ee6ea263181'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:18.700096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4ef1fdb1-4904-4c2f-92cf-9ee6ea263181 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6416e876babe4a30b21eaaa002360013', 'request_id': 'req-4ef1fdb1-4904-4c2f-92cf-9ee6ea263181'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:18.706464 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4ef1fdb1-4904-4c2f-92cf-9ee6ea263181 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6416e876babe4a30b21eaaa002360013', 'request_id': 'req-4ef1fdb1-4904-4c2f-92cf-9ee6ea263181'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:18.718413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4ef1fdb1-4904-4c2f-92cf-9ee6ea263181 admin admin] The token cache is being invalidate because project 6416e876babe4a30b21eaaa002360013 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:26:18.718702 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4ef1fdb1-4904-4c2f-92cf-9ee6ea263181 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:18.719425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4ef1fdb1-4904-4c2f-92cf-9ee6ea263181 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:18.720883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1362/2735] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:26:18 2022] DELETE /identity/v3/projects/6416e876babe4a30b21eaaa002360013 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:18.725917 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436478ba-a8f7-434e-a2cb-72f64dafe523 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:18.726401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436478ba-a8f7-434e-a2cb-72f64dafe523 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:18.726498 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436478ba-a8f7-434e-a2cb-72f64dafe523 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:18.746038 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-436478ba-a8f7-434e-a2cb-72f64dafe523 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:18.747226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-436478ba-a8f7-434e-a2cb-72f64dafe523 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:18.804113 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1374/2736] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:18.812458 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c43c8ba-4180-44ef-88fd-79c792899625 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:18.812744 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c43c8ba-4180-44ef-88fd-79c792899625 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:18.813018 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c43c8ba-4180-44ef-88fd-79c792899625 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:18.833934 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-9c43c8ba-4180-44ef-88fd-79c792899625 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:18.835180 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9c43c8ba-4180-44ef-88fd-79c792899625 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:18.842972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1363/2737] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:18.847091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436478ba-a8f7-434e-a2cb-72f64dafe523 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:18.856457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436478ba-a8f7-434e-a2cb-72f64dafe523 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:18.860930 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524afbc7-210b-43f1-b7d1-b1cd5ff8a09a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:18.861512 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524afbc7-210b-43f1-b7d1-b1cd5ff8a09a admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:18.861788 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524afbc7-210b-43f1-b7d1-b1cd5ff8a09a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:18.862056 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524afbc7-210b-43f1-b7d1-b1cd5ff8a09a admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:18.862685 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524afbc7-210b-43f1-b7d1-b1cd5ff8a09a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:26:18.863021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524afbc7-210b-43f1-b7d1-b1cd5ff8a09a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:18.865086 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524afbc7-210b-43f1-b7d1-b1cd5ff8a09a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:18.876512 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1375/2738] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:26:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:26:18.881409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c43c8ba-4180-44ef-88fd-79c792899625 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:18.890519 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c43c8ba-4180-44ef-88fd-79c792899625 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:18.895098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d45107-398c-41e7-b54b-75e3731d3fe5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:18.895690 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d45107-398c-41e7-b54b-75e3731d3fe5 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:18.895965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d45107-398c-41e7-b54b-75e3731d3fe5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:18.896234 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d45107-398c-41e7-b54b-75e3731d3fe5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:18.896852 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d45107-398c-41e7-b54b-75e3731d3fe5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:18.898713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d45107-398c-41e7-b54b-75e3731d3fe5 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:18.925461 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1364/2739] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:26:18 2022] POST /identity/v3/projects => generated 373 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:26:18.930504 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524afbc7-210b-43f1-b7d1-b1cd5ff8a09a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:18.941055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524afbc7-210b-43f1-b7d1-b1cd5ff8a09a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:18.996825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102bb174-5532-4764-b414-f7dd678e5868 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:18.997366 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102bb174-5532-4764-b414-f7dd678e5868 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:18.997650 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102bb174-5532-4764-b414-f7dd678e5868 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:18.997926 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102bb174-5532-4764-b414-f7dd678e5868 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:18.998555 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102bb174-5532-4764-b414-f7dd678e5868 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:19.000517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102bb174-5532-4764-b414-f7dd678e5868 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:19.020864 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1376/2740] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:26:18 2022] POST /identity/v3/users => generated 287 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:26:19.025025 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d45107-398c-41e7-b54b-75e3731d3fe5 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:19.035221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d45107-398c-41e7-b54b-75e3731d3fe5 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:19.040375 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54e7dbb-de72-47a3-bd0c-4ebe17170b8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:19.040966 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54e7dbb-de72-47a3-bd0c-4ebe17170b8a admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:19.041248 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54e7dbb-de72-47a3-bd0c-4ebe17170b8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:19.041520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54e7dbb-de72-47a3-bd0c-4ebe17170b8a admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:19.042152 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54e7dbb-de72-47a3-bd0c-4ebe17170b8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:19.043765 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54e7dbb-de72-47a3-bd0c-4ebe17170b8a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:19.054830 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1365/2741] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:19 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:26:19.058533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102bb174-5532-4764-b414-f7dd678e5868 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:19.068392 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102bb174-5532-4764-b414-f7dd678e5868 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:19.073014 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3c16cf-54c2-4a30-b46b-33c89893cea7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:19.073597 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3c16cf-54c2-4a30-b46b-33c89893cea7 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:19.073901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3c16cf-54c2-4a30-b46b-33c89893cea7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:19.074171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3c16cf-54c2-4a30-b46b-33c89893cea7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:19.074800 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3c16cf-54c2-4a30-b46b-33c89893cea7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:19.076409 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3c16cf-54c2-4a30-b46b-33c89893cea7 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:19.088015 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1377/2742] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:19 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:26:19.091952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54e7dbb-de72-47a3-bd0c-4ebe17170b8a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:19.102518 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54e7dbb-de72-47a3-bd0c-4ebe17170b8a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:19.107174 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60075638-48c4-4f10-9f18-78494a441d52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:19.107851 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60075638-48c4-4f10-9f18-78494a441d52 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:19.108148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60075638-48c4-4f10-9f18-78494a441d52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:19.108442 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60075638-48c4-4f10-9f18-78494a441d52 admin admin] PATH_INFO: `/v3/projects/4905ddaf27904cd0b09439bfa29d0da2/users/28780284707144849e9ccd7c84f292b7/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:19.123602 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60075638-48c4-4f10-9f18-78494a441d52 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4905ddaf27904cd0b09439bfa29d0da2, user_id=28780284707144849e9ccd7c84f292b7, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:19.126196 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60075638-48c4-4f10-9f18-78494a441d52 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:19.138488 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1366/2743] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:26:19 2022] PUT /identity/v3/projects/4905ddaf27904cd0b09439bfa29d0da2/users/28780284707144849e9ccd7c84f292b7/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:19.149602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16726997-232c-469b-80c6-f8fae6f79ed0 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:19.149887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16726997-232c-469b-80c6-f8fae6f79ed0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:19.150159 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16726997-232c-469b-80c6-f8fae6f79ed0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:19.179289 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-16726997-232c-469b-80c6-f8fae6f79ed0 None None] MFA Rules not processed for user `28780284707144849e9ccd7c84f292b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:19.180742 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-16726997-232c-469b-80c6-f8fae6f79ed0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:19.235162 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:19.245243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1378/2744] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:19 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:19.255840 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60075638-48c4-4f10-9f18-78494a441d52 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:19.257520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-60075638-48c4-4f10-9f18-78494a441d52 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:19.313626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60075638-48c4-4f10-9f18-78494a441d52 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:19.317227 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e49f95e-9478-4983-80f0-608a5e76cda7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:19.317818 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e49f95e-9478-4983-80f0-608a5e76cda7 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:19.318094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e49f95e-9478-4983-80f0-608a5e76cda7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:19.318362 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e49f95e-9478-4983-80f0-608a5e76cda7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:19.326523 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e49f95e-9478-4983-80f0-608a5e76cda7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:19.328050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e49f95e-9478-4983-80f0-608a5e76cda7 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:19.337138 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1367/2745] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:19 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:26:29.243128 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:29.332897 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=79,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:30.895629 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16726997-232c-469b-80c6-f8fae6f79ed0 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:30.959966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16726997-232c-469b-80c6-f8fae6f79ed0 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:30.964358 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4eb871d-93b9-44e8-81f1-d8de69519bc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:30.964955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4eb871d-93b9-44e8-81f1-d8de69519bc3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:30.965229 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4eb871d-93b9-44e8-81f1-d8de69519bc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:30.965506 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4eb871d-93b9-44e8-81f1-d8de69519bc3 admin admin] PATH_INFO: `/v3/users/28780284707144849e9ccd7c84f292b7` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:30.966606 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4eb871d-93b9-44e8-81f1-d8de69519bc3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=28780284707144849e9ccd7c84f292b7)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:30.968532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4eb871d-93b9-44e8-81f1-d8de69519bc3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:31.015651 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c4eb871d-93b9-44e8-81f1-d8de69519bc3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '28780284707144849e9ccd7c84f292b7'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:31.016317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c4eb871d-93b9-44e8-81f1-d8de69519bc3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '28780284707144849e9ccd7c84f292b7'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:31.025036 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c4eb871d-93b9-44e8-81f1-d8de69519bc3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '28780284707144849e9ccd7c84f292b7'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:31.025642 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c4eb871d-93b9-44e8-81f1-d8de69519bc3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '28780284707144849e9ccd7c84f292b7'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:31.038073 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c4eb871d-93b9-44e8-81f1-d8de69519bc3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '28780284707144849e9ccd7c84f292b7'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:31.050839 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1379/2746] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:26:30 2022] DELETE /identity/v3/users/28780284707144849e9ccd7c84f292b7 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:31.056002 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3842aa3a-79d1-4b05-8355-411ce1a77374 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:31.056451 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3842aa3a-79d1-4b05-8355-411ce1a77374 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:31.056545 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3842aa3a-79d1-4b05-8355-411ce1a77374 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:31.075191 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-3842aa3a-79d1-4b05-8355-411ce1a77374 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:31.076385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3842aa3a-79d1-4b05-8355-411ce1a77374 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:31.141243 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1368/2747] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:26:31.151183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4eb871d-93b9-44e8-81f1-d8de69519bc3 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:31.152428 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c4eb871d-93b9-44e8-81f1-d8de69519bc3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:31.213189 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4eb871d-93b9-44e8-81f1-d8de69519bc3 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:31.218347 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefb5879-9a4a-42cc-bb92-120596e8f887 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:31.219008 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefb5879-9a4a-42cc-bb92-120596e8f887 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:31.219279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefb5879-9a4a-42cc-bb92-120596e8f887 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:31.219545 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefb5879-9a4a-42cc-bb92-120596e8f887 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:31.230549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefb5879-9a4a-42cc-bb92-120596e8f887 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:31.231994 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefb5879-9a4a-42cc-bb92-120596e8f887 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:31.246448 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1380/2748] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:26:31.519113 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3842aa3a-79d1-4b05-8355-411ce1a77374 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:31.521130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3842aa3a-79d1-4b05-8355-411ce1a77374 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:31.540381 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3842aa3a-79d1-4b05-8355-411ce1a77374 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:31.545951 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf46b910-12d3-49f2-91ec-23bd51f0ed21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:31.546588 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf46b910-12d3-49f2-91ec-23bd51f0ed21 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:31.546896 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf46b910-12d3-49f2-91ec-23bd51f0ed21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:31.547173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf46b910-12d3-49f2-91ec-23bd51f0ed21 admin admin] PATH_INFO: `/v3/projects/4905ddaf27904cd0b09439bfa29d0da2` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:31.548286 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf46b910-12d3-49f2-91ec-23bd51f0ed21 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4905ddaf27904cd0b09439bfa29d0da2)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:31.550175 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf46b910-12d3-49f2-91ec-23bd51f0ed21 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:31.596963 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bf46b910-12d3-49f2-91ec-23bd51f0ed21 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4905ddaf27904cd0b09439bfa29d0da2', 'request_id': 'req-bf46b910-12d3-49f2-91ec-23bd51f0ed21'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:31.603221 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bf46b910-12d3-49f2-91ec-23bd51f0ed21 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4905ddaf27904cd0b09439bfa29d0da2', 'request_id': 'req-bf46b910-12d3-49f2-91ec-23bd51f0ed21'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:31.609583 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bf46b910-12d3-49f2-91ec-23bd51f0ed21 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4905ddaf27904cd0b09439bfa29d0da2', 'request_id': 'req-bf46b910-12d3-49f2-91ec-23bd51f0ed21'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:31.622553 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bf46b910-12d3-49f2-91ec-23bd51f0ed21 admin admin] The token cache is being invalidate because project 4905ddaf27904cd0b09439bfa29d0da2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:26:31.623014 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bf46b910-12d3-49f2-91ec-23bd51f0ed21 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:31.623624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-bf46b910-12d3-49f2-91ec-23bd51f0ed21 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:31.625110 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1369/2749] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:26:31 2022] DELETE /identity/v3/projects/4905ddaf27904cd0b09439bfa29d0da2 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:26:31.631177 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6596a605-1edd-4ed8-a679-0bc751d37e62 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:31.631441 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6596a605-1edd-4ed8-a679-0bc751d37e62 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:31.631710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6596a605-1edd-4ed8-a679-0bc751d37e62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:31.651724 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-6596a605-1edd-4ed8-a679-0bc751d37e62 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:31.652837 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6596a605-1edd-4ed8-a679-0bc751d37e62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:31.669756 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:31.714179 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1381/2750] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:31.721753 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de2a826-0742-4b20-955b-229ae5e35f5a None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:31.722040 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de2a826-0742-4b20-955b-229ae5e35f5a None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:31.722318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de2a826-0742-4b20-955b-229ae5e35f5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:31.742070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-2de2a826-0742-4b20-955b-229ae5e35f5a None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:31.743310 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2de2a826-0742-4b20-955b-229ae5e35f5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:31.751485 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1370/2751] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:31.755895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6596a605-1edd-4ed8-a679-0bc751d37e62 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:31.766248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6596a605-1edd-4ed8-a679-0bc751d37e62 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:31.770966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efb5633-f80e-4971-abc7-5d8e4fafe614 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:31.771559 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efb5633-f80e-4971-abc7-5d8e4fafe614 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:31.771840 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efb5633-f80e-4971-abc7-5d8e4fafe614 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:31.772109 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efb5633-f80e-4971-abc7-5d8e4fafe614 admin admin] PATH_INFO: `/v3/domains` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:31.772746 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efb5633-f80e-4971-abc7-5d8e4fafe614 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81405) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:26:31.773051 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efb5633-f80e-4971-abc7-5d8e4fafe614 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:31.774613 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efb5633-f80e-4971-abc7-5d8e4fafe614 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:31.787299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1382/2752] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:26:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:26:31.791660 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de2a826-0742-4b20-955b-229ae5e35f5a None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:31.801964 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de2a826-0742-4b20-955b-229ae5e35f5a None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:31.806794 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9d3e55-e51d-46d7-9c89-9b5860fb48b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:31.807412 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9d3e55-e51d-46d7-9c89-9b5860fb48b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:31.807685 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9d3e55-e51d-46d7-9c89-9b5860fb48b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:31.807951 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9d3e55-e51d-46d7-9c89-9b5860fb48b2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:31.808603 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9d3e55-e51d-46d7-9c89-9b5860fb48b2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:31.810461 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9d3e55-e51d-46d7-9c89-9b5860fb48b2 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:31.836996 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1371/2753] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:26:31 2022] POST /identity/v3/projects => generated 369 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:26:31.841055 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efb5633-f80e-4971-abc7-5d8e4fafe614 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:31.851483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efb5633-f80e-4971-abc7-5d8e4fafe614 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:31.906705 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a3e5aa-8d4b-4866-804b-de205d6cb521 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:31.907341 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a3e5aa-8d4b-4866-804b-de205d6cb521 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:31.907531 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a3e5aa-8d4b-4866-804b-de205d6cb521 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:31.907804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a3e5aa-8d4b-4866-804b-de205d6cb521 admin admin] PATH_INFO: `/v3/users` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:31.908435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a3e5aa-8d4b-4866-804b-de205d6cb521 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:31.910833 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a3e5aa-8d4b-4866-804b-de205d6cb521 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:31.932446 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1383/2754] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:26:31 2022] POST /identity/v3/users => generated 285 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:26:31.936455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9d3e55-e51d-46d7-9c89-9b5860fb48b2 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:31.947246 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9d3e55-e51d-46d7-9c89-9b5860fb48b2 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:31.951859 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6db1b85-1fea-4558-b9b5-cb0972562c31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:31.952453 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6db1b85-1fea-4558-b9b5-cb0972562c31 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:31.952733 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6db1b85-1fea-4558-b9b5-cb0972562c31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:31.953005 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6db1b85-1fea-4558-b9b5-cb0972562c31 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:31.953639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6db1b85-1fea-4558-b9b5-cb0972562c31 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:31.955303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6db1b85-1fea-4558-b9b5-cb0972562c31 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:31.965178 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1372/2755] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:31 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:26:31.969068 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a3e5aa-8d4b-4866-804b-de205d6cb521 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:31.979470 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a3e5aa-8d4b-4866-804b-de205d6cb521 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:31.984020 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec62bc04-3200-4262-aac8-87673bcfa122 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:31.984601 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec62bc04-3200-4262-aac8-87673bcfa122 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:31.984879 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec62bc04-3200-4262-aac8-87673bcfa122 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:31.985149 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec62bc04-3200-4262-aac8-87673bcfa122 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:31.985779 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec62bc04-3200-4262-aac8-87673bcfa122 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:31.987378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec62bc04-3200-4262-aac8-87673bcfa122 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:31.997713 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1384/2756] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:31 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:26:32.001604 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6db1b85-1fea-4558-b9b5-cb0972562c31 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:32.011971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6db1b85-1fea-4558-b9b5-cb0972562c31 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:32.016492 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe32ef46-4fcd-4e73-b98c-9cc45a01214c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:32.017126 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe32ef46-4fcd-4e73-b98c-9cc45a01214c admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:32.017405 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe32ef46-4fcd-4e73-b98c-9cc45a01214c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:32.017678 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe32ef46-4fcd-4e73-b98c-9cc45a01214c admin admin] PATH_INFO: `/v3/projects/9fca31eaad0a453b9ee50c73fd5b99a5/users/82e4e65638e54903acd34afa87f47505/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:32.031971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe32ef46-4fcd-4e73-b98c-9cc45a01214c admin admin] RBAC: Authorizing `identity:create_grant(project_id=9fca31eaad0a453b9ee50c73fd5b99a5, user_id=82e4e65638e54903acd34afa87f47505, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:32.034574 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe32ef46-4fcd-4e73-b98c-9cc45a01214c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:32.047632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1373/2757] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:26:32 2022] PUT /identity/v3/projects/9fca31eaad0a453b9ee50c73fd5b99a5/users/82e4e65638e54903acd34afa87f47505/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:32.058383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f24f44-73e8-422a-b9ea-88d75dc2ef9e None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:32.058682 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f24f44-73e8-422a-b9ea-88d75dc2ef9e None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:32.058986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f24f44-73e8-422a-b9ea-88d75dc2ef9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:32.086565 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-43f24f44-73e8-422a-b9ea-88d75dc2ef9e None None] MFA Rules not processed for user `82e4e65638e54903acd34afa87f47505`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:32.087821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-43f24f44-73e8-422a-b9ea-88d75dc2ef9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:32.152231 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1385/2758] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:32 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:26:32.164385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe32ef46-4fcd-4e73-b98c-9cc45a01214c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:32.166609 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fe32ef46-4fcd-4e73-b98c-9cc45a01214c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:32.188709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:32.228283 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe32ef46-4fcd-4e73-b98c-9cc45a01214c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:32.231974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69c9b5b-75c6-4edd-b3be-cbcfa00e778a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eef98bcf2699461d9a7f3c3bffe82f34', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:32.232577 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69c9b5b-75c6-4edd-b3be-cbcfa00e778a service glance] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:32.232856 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69c9b5b-75c6-4edd-b3be-cbcfa00e778a service glance] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:32.233129 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69c9b5b-75c6-4edd-b3be-cbcfa00e778a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:32.241417 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69c9b5b-75c6-4edd-b3be-cbcfa00e778a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:32.242835 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69c9b5b-75c6-4edd-b3be-cbcfa00e778a service glance] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:32.251539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1374/2759] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:32 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:26:37.783680 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f24f44-73e8-422a-b9ea-88d75dc2ef9e None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:37.785605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-43f24f44-73e8-422a-b9ea-88d75dc2ef9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:37.807187 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:37.919456 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f24f44-73e8-422a-b9ea-88d75dc2ef9e None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:37.924037 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9619d7-6c77-4908-b839-b7ffc60fef9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:37.924664 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9619d7-6c77-4908-b839-b7ffc60fef9a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:37.924939 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9619d7-6c77-4908-b839-b7ffc60fef9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:37.925206 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9619d7-6c77-4908-b839-b7ffc60fef9a admin admin] PATH_INFO: `/v3/users/da3fd9b1ea02427fb371cf19affd12b8` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:37.926372 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9619d7-6c77-4908-b839-b7ffc60fef9a admin admin] RBAC: Authorizing `identity:delete_user(user_id=da3fd9b1ea02427fb371cf19affd12b8)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:37.928289 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9619d7-6c77-4908-b839-b7ffc60fef9a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:37.980102 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ee9619d7-6c77-4908-b839-b7ffc60fef9a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'da3fd9b1ea02427fb371cf19affd12b8'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:37.980896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ee9619d7-6c77-4908-b839-b7ffc60fef9a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'da3fd9b1ea02427fb371cf19affd12b8'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:37.988532 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ee9619d7-6c77-4908-b839-b7ffc60fef9a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'da3fd9b1ea02427fb371cf19affd12b8'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:37.989375 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ee9619d7-6c77-4908-b839-b7ffc60fef9a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'da3fd9b1ea02427fb371cf19affd12b8'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:38.001309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-ee9619d7-6c77-4908-b839-b7ffc60fef9a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'da3fd9b1ea02427fb371cf19affd12b8'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:38.014081 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1386/2760] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:26:37 2022] DELETE /identity/v3/users/da3fd9b1ea02427fb371cf19affd12b8 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:38.018617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b86da9a-e289-4ad7-a815-f1132750eb71 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:38.019116 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b86da9a-e289-4ad7-a815-f1132750eb71 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:38.019223 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b86da9a-e289-4ad7-a815-f1132750eb71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:38.047585 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-5b86da9a-e289-4ad7-a815-f1132750eb71 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:38.048782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5b86da9a-e289-4ad7-a815-f1132750eb71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:38.111888 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1375/2761] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:38.121714 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9619d7-6c77-4908-b839-b7ffc60fef9a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:38.123181 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ee9619d7-6c77-4908-b839-b7ffc60fef9a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:38.167342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:38.203871 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9619d7-6c77-4908-b839-b7ffc60fef9a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:38.207483 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c193a6c-c338-4fa2-8e15-a88694f4f705 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:38.208085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c193a6c-c338-4fa2-8e15-a88694f4f705 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:38.208365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c193a6c-c338-4fa2-8e15-a88694f4f705 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:38.208641 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c193a6c-c338-4fa2-8e15-a88694f4f705 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:38.220380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c193a6c-c338-4fa2-8e15-a88694f4f705 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:38.220380 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c193a6c-c338-4fa2-8e15-a88694f4f705 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:38.230696 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1387/2762] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:26:38.509205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b86da9a-e289-4ad7-a815-f1132750eb71 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:38.511864 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5b86da9a-e289-4ad7-a815-f1132750eb71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:38.538304 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b86da9a-e289-4ad7-a815-f1132750eb71 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:38.543368 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0548f4c-da4d-4db4-9e37-c590da23730d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:38.544033 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0548f4c-da4d-4db4-9e37-c590da23730d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:38.544310 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0548f4c-da4d-4db4-9e37-c590da23730d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:38.544582 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0548f4c-da4d-4db4-9e37-c590da23730d admin admin] PATH_INFO: `/v3/projects/800746745cc74a04ae43f97a17f1a722` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:38.545733 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0548f4c-da4d-4db4-9e37-c590da23730d admin admin] RBAC: Authorizing `identity:delete_project(project_id=800746745cc74a04ae43f97a17f1a722)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:38.547664 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0548f4c-da4d-4db4-9e37-c590da23730d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:38.596744 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c0548f4c-da4d-4db4-9e37-c590da23730d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '800746745cc74a04ae43f97a17f1a722', 'request_id': 'req-c0548f4c-da4d-4db4-9e37-c590da23730d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:38.603382 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c0548f4c-da4d-4db4-9e37-c590da23730d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '800746745cc74a04ae43f97a17f1a722', 'request_id': 'req-c0548f4c-da4d-4db4-9e37-c590da23730d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:38.610353 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c0548f4c-da4d-4db4-9e37-c590da23730d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '800746745cc74a04ae43f97a17f1a722', 'request_id': 'req-c0548f4c-da4d-4db4-9e37-c590da23730d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:38.626548 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c0548f4c-da4d-4db4-9e37-c590da23730d admin admin] The token cache is being invalidate because project 800746745cc74a04ae43f97a17f1a722 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:26:38.626844 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c0548f4c-da4d-4db4-9e37-c590da23730d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:38.627665 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-c0548f4c-da4d-4db4-9e37-c590da23730d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:38.629385 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1376/2763] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:26:38 2022] DELETE /identity/v3/projects/800746745cc74a04ae43f97a17f1a722 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:47.150612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c193a6c-c338-4fa2-8e15-a88694f4f705 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:47.152210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6c193a6c-c338-4fa2-8e15-a88694f4f705 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:47.251921 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0548f4c-da4d-4db4-9e37-c590da23730d admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:47.253449 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-c0548f4c-da4d-4db4-9e37-c590da23730d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:47.253960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c193a6c-c338-4fa2-8e15-a88694f4f705 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:47.298069 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:47.315289 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c2e7be-35fc-4ec7-8010-b338614aef35 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8eaff1b843a145a49994dcf089f44315', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:47.316004 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c2e7be-35fc-4ec7-8010-b338614aef35 service placement] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:47.316286 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c2e7be-35fc-4ec7-8010-b338614aef35 service placement] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:47.316558 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c2e7be-35fc-4ec7-8010-b338614aef35 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:47.318802 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e2c2e7be-35fc-4ec7-8010-b338614aef35 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:47.321058 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0548f4c-da4d-4db4-9e37-c590da23730d admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:47.325723 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fa3c25-291e-4bbb-887a-da3094acbc1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:47.326396 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fa3c25-291e-4bbb-887a-da3094acbc1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:47.326674 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fa3c25-291e-4bbb-887a-da3094acbc1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:47.326982 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fa3c25-291e-4bbb-887a-da3094acbc1d admin admin] PATH_INFO: `/v3/users/c2dd1fba00284664b4572d394143f81e` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:47.328151 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fa3c25-291e-4bbb-887a-da3094acbc1d admin admin] RBAC: Authorizing `identity:delete_user(user_id=c2dd1fba00284664b4572d394143f81e)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:47.330094 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fa3c25-291e-4bbb-887a-da3094acbc1d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:47.330212 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c2e7be-35fc-4ec7-8010-b338614aef35 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:47.331751 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c2e7be-35fc-4ec7-8010-b338614aef35 service placement] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:47.342995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1388/2764] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:47 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:26:47.378178 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e8fa3c25-291e-4bbb-887a-da3094acbc1d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c2dd1fba00284664b4572d394143f81e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:47.378870 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e8fa3c25-291e-4bbb-887a-da3094acbc1d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c2dd1fba00284664b4572d394143f81e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:47.387793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e8fa3c25-291e-4bbb-887a-da3094acbc1d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c2dd1fba00284664b4572d394143f81e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:47.388401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e8fa3c25-291e-4bbb-887a-da3094acbc1d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c2dd1fba00284664b4572d394143f81e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:47.396553 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-e8fa3c25-291e-4bbb-887a-da3094acbc1d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c2dd1fba00284664b4572d394143f81e'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:47.425031 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1377/2765] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:26:47 2022] DELETE /identity/v3/users/c2dd1fba00284664b4572d394143f81e => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:26:47.429765 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b8301d-9617-445b-9aa1-2ce1d2b6a469 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:47.430076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b8301d-9617-445b-9aa1-2ce1d2b6a469 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:47.430352 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b8301d-9617-445b-9aa1-2ce1d2b6a469 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:47.452708 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-81b8301d-9617-445b-9aa1-2ce1d2b6a469 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:47.454450 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-81b8301d-9617-445b-9aa1-2ce1d2b6a469 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:47.487077 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:47.517699 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1389/2766] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:47.527437 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fa3c25-291e-4bbb-887a-da3094acbc1d admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:47.528918 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-e8fa3c25-291e-4bbb-887a-da3094acbc1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:47.585009 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fa3c25-291e-4bbb-887a-da3094acbc1d admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:47.588631 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac252592-660d-4689-a65c-b8192e713290 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:47.589225 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac252592-660d-4689-a65c-b8192e713290 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:47.589501 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac252592-660d-4689-a65c-b8192e713290 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:47.589773 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac252592-660d-4689-a65c-b8192e713290 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:47.598505 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac252592-660d-4689-a65c-b8192e713290 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:47.599987 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac252592-660d-4689-a65c-b8192e713290 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:47.609161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1378/2767] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:26:47.870835 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b8301d-9617-445b-9aa1-2ce1d2b6a469 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:47.872323 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-81b8301d-9617-445b-9aa1-2ce1d2b6a469 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:47.887362 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b8301d-9617-445b-9aa1-2ce1d2b6a469 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:47.892248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca1b8d0-b1bb-4753-85f5-9c18fadbd71c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:47.892886 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca1b8d0-b1bb-4753-85f5-9c18fadbd71c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:47.893166 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca1b8d0-b1bb-4753-85f5-9c18fadbd71c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:47.893462 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca1b8d0-b1bb-4753-85f5-9c18fadbd71c admin admin] PATH_INFO: `/v3/projects/cce68dff03de4de5be79b5425b7b8bbe` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:47.894609 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca1b8d0-b1bb-4753-85f5-9c18fadbd71c admin admin] RBAC: Authorizing `identity:delete_project(project_id=cce68dff03de4de5be79b5425b7b8bbe)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:47.896533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca1b8d0-b1bb-4753-85f5-9c18fadbd71c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:47.943489 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5ca1b8d0-b1bb-4753-85f5-9c18fadbd71c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cce68dff03de4de5be79b5425b7b8bbe', 'request_id': 'req-5ca1b8d0-b1bb-4753-85f5-9c18fadbd71c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:47.949511 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5ca1b8d0-b1bb-4753-85f5-9c18fadbd71c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cce68dff03de4de5be79b5425b7b8bbe', 'request_id': 'req-5ca1b8d0-b1bb-4753-85f5-9c18fadbd71c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:47.955760 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5ca1b8d0-b1bb-4753-85f5-9c18fadbd71c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cce68dff03de4de5be79b5425b7b8bbe', 'request_id': 'req-5ca1b8d0-b1bb-4753-85f5-9c18fadbd71c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:47.968329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5ca1b8d0-b1bb-4753-85f5-9c18fadbd71c admin admin] The token cache is being invalidate because project cce68dff03de4de5be79b5425b7b8bbe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:26:47.968624 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5ca1b8d0-b1bb-4753-85f5-9c18fadbd71c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:47.969245 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5ca1b8d0-b1bb-4753-85f5-9c18fadbd71c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:47.970696 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1390/2768] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:26:47 2022] DELETE /identity/v3/projects/cce68dff03de4de5be79b5425b7b8bbe => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:47.975302 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac252592-660d-4689-a65c-b8192e713290 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:47.984782 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac252592-660d-4689-a65c-b8192e713290 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:48.040077 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65631e28-2166-473b-981c-77ffbc204880 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:48.040712 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65631e28-2166-473b-981c-77ffbc204880 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:48.040992 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65631e28-2166-473b-981c-77ffbc204880 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:48.041249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65631e28-2166-473b-981c-77ffbc204880 admin admin] PATH_INFO: `/v3/users/1ac1bcade40a4d5f91aeb2cdb29572d2` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:48.042346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65631e28-2166-473b-981c-77ffbc204880 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1ac1bcade40a4d5f91aeb2cdb29572d2)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:48.044307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65631e28-2166-473b-981c-77ffbc204880 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:48.046592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:48.092874 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-65631e28-2166-473b-981c-77ffbc204880 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1ac1bcade40a4d5f91aeb2cdb29572d2'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:48.093516 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-65631e28-2166-473b-981c-77ffbc204880 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1ac1bcade40a4d5f91aeb2cdb29572d2'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:48.101150 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-65631e28-2166-473b-981c-77ffbc204880 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1ac1bcade40a4d5f91aeb2cdb29572d2'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:48.101794 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-65631e28-2166-473b-981c-77ffbc204880 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1ac1bcade40a4d5f91aeb2cdb29572d2'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:48.108580 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-65631e28-2166-473b-981c-77ffbc204880 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1ac1bcade40a4d5f91aeb2cdb29572d2'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:48.126147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1379/2769] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:26:47 2022] DELETE /identity/v3/users/1ac1bcade40a4d5f91aeb2cdb29572d2 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:48.400194 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca1b8d0-b1bb-4753-85f5-9c18fadbd71c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:48.401743 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-5ca1b8d0-b1bb-4753-85f5-9c18fadbd71c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:48.415859 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca1b8d0-b1bb-4753-85f5-9c18fadbd71c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:48.473255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5638cbab-c019-4e95-bcf3-efaf6f86d4ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:48.474414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5638cbab-c019-4e95-bcf3-efaf6f86d4ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:48.474689 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5638cbab-c019-4e95-bcf3-efaf6f86d4ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:48.474987 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5638cbab-c019-4e95-bcf3-efaf6f86d4ac admin admin] PATH_INFO: `/v3/projects/26d09e908be540a3a7e1b7548e735766` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:48.476103 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5638cbab-c019-4e95-bcf3-efaf6f86d4ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=26d09e908be540a3a7e1b7548e735766)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:48.478006 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5638cbab-c019-4e95-bcf3-efaf6f86d4ac admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:48.526895 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5638cbab-c019-4e95-bcf3-efaf6f86d4ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '26d09e908be540a3a7e1b7548e735766', 'request_id': 'req-5638cbab-c019-4e95-bcf3-efaf6f86d4ac'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:48.531807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5638cbab-c019-4e95-bcf3-efaf6f86d4ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '26d09e908be540a3a7e1b7548e735766', 'request_id': 'req-5638cbab-c019-4e95-bcf3-efaf6f86d4ac'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:48.533686 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:48.542522 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5638cbab-c019-4e95-bcf3-efaf6f86d4ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '26d09e908be540a3a7e1b7548e735766', 'request_id': 'req-5638cbab-c019-4e95-bcf3-efaf6f86d4ac'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:48.558174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5638cbab-c019-4e95-bcf3-efaf6f86d4ac admin admin] The token cache is being invalidate because project 26d09e908be540a3a7e1b7548e735766 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:26:48.558467 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5638cbab-c019-4e95-bcf3-efaf6f86d4ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:48.559126 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-5638cbab-c019-4e95-bcf3-efaf6f86d4ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:48.560618 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1391/2770] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:26:48 2022] DELETE /identity/v3/projects/26d09e908be540a3a7e1b7548e735766 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:48.565533 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1735821b-9559-4d78-988f-054484d237d5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:48.566046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1735821b-9559-4d78-988f-054484d237d5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:48.566154 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1735821b-9559-4d78-988f-054484d237d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:48.614329 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-1735821b-9559-4d78-988f-054484d237d5 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:48.615592 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1735821b-9559-4d78-988f-054484d237d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:48.673521 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1380/2771] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:48.682062 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7854bebc-8659-4d6b-a103-26842b59c2d0 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:48.682351 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7854bebc-8659-4d6b-a103-26842b59c2d0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:48.682630 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7854bebc-8659-4d6b-a103-26842b59c2d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:48.702703 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7854bebc-8659-4d6b-a103-26842b59c2d0 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:48.703930 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7854bebc-8659-4d6b-a103-26842b59c2d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:48.712384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1392/2772] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:48.716443 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1735821b-9559-4d78-988f-054484d237d5 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:48.725908 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1735821b-9559-4d78-988f-054484d237d5 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:48.730466 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3fb269-ea63-4567-a0f5-771bd5e8058a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:48.731027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3fb269-ea63-4567-a0f5-771bd5e8058a admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:48.731305 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3fb269-ea63-4567-a0f5-771bd5e8058a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:48.731573 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3fb269-ea63-4567-a0f5-771bd5e8058a admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:48.732204 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3fb269-ea63-4567-a0f5-771bd5e8058a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:26:48.732531 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3fb269-ea63-4567-a0f5-771bd5e8058a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:48.734096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3fb269-ea63-4567-a0f5-771bd5e8058a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:48.746609 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1381/2773] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:26:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:26:48.751526 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7854bebc-8659-4d6b-a103-26842b59c2d0 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:48.761466 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7854bebc-8659-4d6b-a103-26842b59c2d0 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:48.766005 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abd5497-a2f2-4162-b41d-240c6c8e3953 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:48.766592 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abd5497-a2f2-4162-b41d-240c6c8e3953 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:48.766945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abd5497-a2f2-4162-b41d-240c6c8e3953 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:48.767157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abd5497-a2f2-4162-b41d-240c6c8e3953 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:48.767785 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abd5497-a2f2-4162-b41d-240c6c8e3953 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:48.769641 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abd5497-a2f2-4162-b41d-240c6c8e3953 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:48.796804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1393/2774] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:26:48 2022] POST /identity/v3/projects => generated 399 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:26:48.801071 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3fb269-ea63-4567-a0f5-771bd5e8058a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:48.811300 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3fb269-ea63-4567-a0f5-771bd5e8058a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:48.853169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:48.870743 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903ead9c-d5f3-4bb5-8d67-9994ce5ee0d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:48.871379 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903ead9c-d5f3-4bb5-8d67-9994ce5ee0d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:48.871661 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903ead9c-d5f3-4bb5-8d67-9994ce5ee0d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:48.871934 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903ead9c-d5f3-4bb5-8d67-9994ce5ee0d9 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:48.872570 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903ead9c-d5f3-4bb5-8d67-9994ce5ee0d9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:48.874470 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903ead9c-d5f3-4bb5-8d67-9994ce5ee0d9 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:48.896254 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1382/2775] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:26:48 2022] POST /identity/v3/users => generated 300 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:26:48.900382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abd5497-a2f2-4162-b41d-240c6c8e3953 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:48.910469 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abd5497-a2f2-4162-b41d-240c6c8e3953 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:48.915261 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69a9fba-47ae-4a8f-bf23-465e9b2a4fca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:48.915873 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69a9fba-47ae-4a8f-bf23-465e9b2a4fca admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:48.916153 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69a9fba-47ae-4a8f-bf23-465e9b2a4fca admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:48.916420 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69a9fba-47ae-4a8f-bf23-465e9b2a4fca admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:48.917052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69a9fba-47ae-4a8f-bf23-465e9b2a4fca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:48.918701 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69a9fba-47ae-4a8f-bf23-465e9b2a4fca admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:48.928652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1394/2776] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:48 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:26:48.933236 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903ead9c-d5f3-4bb5-8d67-9994ce5ee0d9 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:48.943678 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903ead9c-d5f3-4bb5-8d67-9994ce5ee0d9 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:48.948308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c72e097-fcd4-4c01-a389-97c22de22053 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:48.948906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c72e097-fcd4-4c01-a389-97c22de22053 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:48.949184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c72e097-fcd4-4c01-a389-97c22de22053 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:48.949455 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c72e097-fcd4-4c01-a389-97c22de22053 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:48.950084 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c72e097-fcd4-4c01-a389-97c22de22053 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:48.951698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c72e097-fcd4-4c01-a389-97c22de22053 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:48.962723 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1383/2777] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:48 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:26:48.967045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69a9fba-47ae-4a8f-bf23-465e9b2a4fca admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:48.976742 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69a9fba-47ae-4a8f-bf23-465e9b2a4fca admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:48.981265 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74f0521-1041-48fe-9a6e-58937cfa3402 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:48.981842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74f0521-1041-48fe-9a6e-58937cfa3402 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:48.982116 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74f0521-1041-48fe-9a6e-58937cfa3402 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:48.982384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74f0521-1041-48fe-9a6e-58937cfa3402 admin admin] PATH_INFO: `/v3/projects/6b170fb0f82945a198d1425d8202e66d/users/7a3ecf3c614a495985f996064ee0cffb/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:48.997028 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74f0521-1041-48fe-9a6e-58937cfa3402 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6b170fb0f82945a198d1425d8202e66d, user_id=7a3ecf3c614a495985f996064ee0cffb, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:48.999684 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74f0521-1041-48fe-9a6e-58937cfa3402 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:49.013052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1395/2778] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:26:48 2022] PUT /identity/v3/projects/6b170fb0f82945a198d1425d8202e66d/users/7a3ecf3c614a495985f996064ee0cffb/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:49.023211 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9dc0fd-a53c-464d-b225-78eb06370073 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:49.023495 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9dc0fd-a53c-464d-b225-78eb06370073 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:49.023765 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9dc0fd-a53c-464d-b225-78eb06370073 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:49.050441 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8b9dc0fd-a53c-464d-b225-78eb06370073 None None] MFA Rules not processed for user `7a3ecf3c614a495985f996064ee0cffb`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:49.052164 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8b9dc0fd-a53c-464d-b225-78eb06370073 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:49.115753 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1384/2779] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:49 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:49.120333 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74f0521-1041-48fe-9a6e-58937cfa3402 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:49.183551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74f0521-1041-48fe-9a6e-58937cfa3402 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:49.188190 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5318ace6-c78b-446c-94a2-003e9c7197a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:49.188847 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5318ace6-c78b-446c-94a2-003e9c7197a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:49.189167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5318ace6-c78b-446c-94a2-003e9c7197a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:49.189470 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5318ace6-c78b-446c-94a2-003e9c7197a6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:49.190260 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5318ace6-c78b-446c-94a2-003e9c7197a6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:49.192830 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5318ace6-c78b-446c-94a2-003e9c7197a6 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:49.224219 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1396/2780] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:26:49 2022] POST /identity/v3/projects => generated 397 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:26:49.229500 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9dc0fd-a53c-464d-b225-78eb06370073 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:49.241589 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9dc0fd-a53c-464d-b225-78eb06370073 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:49.296798 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49e3f8b-fd3d-44e0-84ab-be9d69127566 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:49.297357 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49e3f8b-fd3d-44e0-84ab-be9d69127566 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:49.297637 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49e3f8b-fd3d-44e0-84ab-be9d69127566 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:49.297960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49e3f8b-fd3d-44e0-84ab-be9d69127566 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:49.298539 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49e3f8b-fd3d-44e0-84ab-be9d69127566 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:49.300476 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49e3f8b-fd3d-44e0-84ab-be9d69127566 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:49.321492 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1385/2781] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:26:49 2022] POST /identity/v3/users => generated 305 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:26:49.325567 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5318ace6-c78b-446c-94a2-003e9c7197a6 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:49.335860 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5318ace6-c78b-446c-94a2-003e9c7197a6 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:49.340390 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9f2229-6441-49dd-a6b9-0457f4c875a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:49.340984 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9f2229-6441-49dd-a6b9-0457f4c875a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:49.341264 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9f2229-6441-49dd-a6b9-0457f4c875a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:49.341536 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9f2229-6441-49dd-a6b9-0457f4c875a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:49.342160 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9f2229-6441-49dd-a6b9-0457f4c875a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:49.343755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9f2229-6441-49dd-a6b9-0457f4c875a5 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:49.354557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1397/2782] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:49 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:26:49.358672 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49e3f8b-fd3d-44e0-84ab-be9d69127566 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:49.369097 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49e3f8b-fd3d-44e0-84ab-be9d69127566 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:49.373595 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ecd5a1-2972-47bb-916c-9ea8b63a9a10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:49.374591 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ecd5a1-2972-47bb-916c-9ea8b63a9a10 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:49.374897 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ecd5a1-2972-47bb-916c-9ea8b63a9a10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:49.375173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ecd5a1-2972-47bb-916c-9ea8b63a9a10 admin admin] PATH_INFO: `/v3/projects/846827d35bd24fbf9ea0124a3c02eb9d/users/56e75840a57243aa91fc24ef69c89e9d/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:49.389852 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ecd5a1-2972-47bb-916c-9ea8b63a9a10 admin admin] RBAC: Authorizing `identity:create_grant(project_id=846827d35bd24fbf9ea0124a3c02eb9d, user_id=56e75840a57243aa91fc24ef69c89e9d, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:49.393308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ecd5a1-2972-47bb-916c-9ea8b63a9a10 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:49.405805 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1386/2783] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:26:49 2022] PUT /identity/v3/projects/846827d35bd24fbf9ea0124a3c02eb9d/users/56e75840a57243aa91fc24ef69c89e9d/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:49.409608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9f2229-6441-49dd-a6b9-0457f4c875a5 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:49.420324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9f2229-6441-49dd-a6b9-0457f4c875a5 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:49.476547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec14f5f-a62e-4b62-91fc-6ecee25a290e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:49.477154 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec14f5f-a62e-4b62-91fc-6ecee25a290e admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:49.477437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec14f5f-a62e-4b62-91fc-6ecee25a290e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:49.477712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec14f5f-a62e-4b62-91fc-6ecee25a290e admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:49.478346 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec14f5f-a62e-4b62-91fc-6ecee25a290e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:49.479970 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec14f5f-a62e-4b62-91fc-6ecee25a290e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:49.481198 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:49.492842 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1398/2784] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:49 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:26:49.496622 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ecd5a1-2972-47bb-916c-9ea8b63a9a10 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:49.507057 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ecd5a1-2972-47bb-916c-9ea8b63a9a10 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:49.511723 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2503f7fa-ab6a-4e6a-911f-ac9ed7ca2aed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:49.512393 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2503f7fa-ab6a-4e6a-911f-ac9ed7ca2aed admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:49.512674 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2503f7fa-ab6a-4e6a-911f-ac9ed7ca2aed admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:49.512950 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2503f7fa-ab6a-4e6a-911f-ac9ed7ca2aed admin admin] PATH_INFO: `/v3/projects/846827d35bd24fbf9ea0124a3c02eb9d/users/56e75840a57243aa91fc24ef69c89e9d/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:49.514841 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2503f7fa-ab6a-4e6a-911f-ac9ed7ca2aed admin admin] RBAC: Authorizing `identity:create_grant(project_id=846827d35bd24fbf9ea0124a3c02eb9d, user_id=56e75840a57243aa91fc24ef69c89e9d, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:49.517477 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2503f7fa-ab6a-4e6a-911f-ac9ed7ca2aed admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:49.527566 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1387/2785] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:26:49 2022] PUT /identity/v3/projects/846827d35bd24fbf9ea0124a3c02eb9d/users/56e75840a57243aa91fc24ef69c89e9d/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:49.538801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7356d8b0-7921-4d0d-9e94-662aba1311cf None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:49.539118 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7356d8b0-7921-4d0d-9e94-662aba1311cf None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:49.539401 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7356d8b0-7921-4d0d-9e94-662aba1311cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:49.568963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7356d8b0-7921-4d0d-9e94-662aba1311cf None None] MFA Rules not processed for user `56e75840a57243aa91fc24ef69c89e9d`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:49.570314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7356d8b0-7921-4d0d-9e94-662aba1311cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:49.639579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1399/2786] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:49 2022] POST /identity/v3/auth/tokens => generated 3341 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:49.650885 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2503f7fa-ab6a-4e6a-911f-ac9ed7ca2aed admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:49.652615 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-2503f7fa-ab6a-4e6a-911f-ac9ed7ca2aed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:49.659320 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:49.714688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2503f7fa-ab6a-4e6a-911f-ac9ed7ca2aed admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:49.718525 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cab5df-4ebc-418d-89f8-3ccbefb52c5f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:49.719182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cab5df-4ebc-418d-89f8-3ccbefb52c5f service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:49.719471 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cab5df-4ebc-418d-89f8-3ccbefb52c5f service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:49.719731 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cab5df-4ebc-418d-89f8-3ccbefb52c5f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:49.728952 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76cab5df-4ebc-418d-89f8-3ccbefb52c5f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:49.730381 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76cab5df-4ebc-418d-89f8-3ccbefb52c5f service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:49.740364 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1388/2787] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:49 2022] GET /identity/v3/auth/tokens => generated 3341 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:26:49.822074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7356d8b0-7921-4d0d-9e94-662aba1311cf None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:49.833113 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7356d8b0-7921-4d0d-9e94-662aba1311cf None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:49.842681 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23371b99-90d8-4b0d-bb24-8fb0d6e43980 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:49.843342 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23371b99-90d8-4b0d-bb24-8fb0d6e43980 service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:49.843670 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23371b99-90d8-4b0d-bb24-8fb0d6e43980 service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:49.843945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23371b99-90d8-4b0d-bb24-8fb0d6e43980 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:49.923966 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23371b99-90d8-4b0d-bb24-8fb0d6e43980 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:49.925802 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23371b99-90d8-4b0d-bb24-8fb0d6e43980 service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:49.942901 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1400/2788] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:49 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:26:57.175977 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cab5df-4ebc-418d-89f8-3ccbefb52c5f service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:57.236501 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cab5df-4ebc-418d-89f8-3ccbefb52c5f service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:57.241443 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13962d2-bffe-43c3-b558-652ffa54a1f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:57.242057 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13962d2-bffe-43c3-b558-652ffa54a1f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:57.242335 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13962d2-bffe-43c3-b558-652ffa54a1f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:57.242607 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13962d2-bffe-43c3-b558-652ffa54a1f9 admin admin] PATH_INFO: `/v3/users/7a3ecf3c614a495985f996064ee0cffb` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:57.243734 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13962d2-bffe-43c3-b558-652ffa54a1f9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7a3ecf3c614a495985f996064ee0cffb)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:57.245626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13962d2-bffe-43c3-b558-652ffa54a1f9 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:57.290630 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a13962d2-bffe-43c3-b558-652ffa54a1f9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7a3ecf3c614a495985f996064ee0cffb'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:57.291229 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a13962d2-bffe-43c3-b558-652ffa54a1f9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7a3ecf3c614a495985f996064ee0cffb'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:57.298693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a13962d2-bffe-43c3-b558-652ffa54a1f9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7a3ecf3c614a495985f996064ee0cffb'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:57.299393 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a13962d2-bffe-43c3-b558-652ffa54a1f9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7a3ecf3c614a495985f996064ee0cffb'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:57.306899 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a13962d2-bffe-43c3-b558-652ffa54a1f9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7a3ecf3c614a495985f996064ee0cffb'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:57.324022 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1389/2789] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:26:57 2022] DELETE /identity/v3/users/7a3ecf3c614a495985f996064ee0cffb => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:26:57.328577 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba19f219-973e-40f9-84ed-6b8cc6cee9b4 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:57.329034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba19f219-973e-40f9-84ed-6b8cc6cee9b4 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:57.329156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba19f219-973e-40f9-84ed-6b8cc6cee9b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:57.347864 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ba19f219-973e-40f9-84ed-6b8cc6cee9b4 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:57.349060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ba19f219-973e-40f9-84ed-6b8cc6cee9b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:57.407329 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1401/2790] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:57.417022 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13962d2-bffe-43c3-b558-652ffa54a1f9 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:57.418409 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a13962d2-bffe-43c3-b558-652ffa54a1f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:57.449092 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:57.497947 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13962d2-bffe-43c3-b558-652ffa54a1f9 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:57.501609 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbefec81-f4ee-4e81-801d-17e395893f63 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:57.502214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbefec81-f4ee-4e81-801d-17e395893f63 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:57.502490 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbefec81-f4ee-4e81-801d-17e395893f63 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:57.502758 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbefec81-f4ee-4e81-801d-17e395893f63 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:57.512224 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbefec81-f4ee-4e81-801d-17e395893f63 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:57.513642 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbefec81-f4ee-4e81-801d-17e395893f63 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:57.522867 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1390/2791] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:26:57.794535 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba19f219-973e-40f9-84ed-6b8cc6cee9b4 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:57.796240 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ba19f219-973e-40f9-84ed-6b8cc6cee9b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:57.809592 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba19f219-973e-40f9-84ed-6b8cc6cee9b4 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:57.814626 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a7827b-b671-459a-aae9-aae432620b2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:57.815318 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a7827b-b671-459a-aae9-aae432620b2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:57.815624 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a7827b-b671-459a-aae9-aae432620b2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:57.815991 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a7827b-b671-459a-aae9-aae432620b2e admin admin] PATH_INFO: `/v3/projects/6b170fb0f82945a198d1425d8202e66d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:57.817629 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a7827b-b671-459a-aae9-aae432620b2e admin admin] RBAC: Authorizing `identity:delete_project(project_id=6b170fb0f82945a198d1425d8202e66d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:57.819582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a7827b-b671-459a-aae9-aae432620b2e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:57.837636 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:57.870080 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-94a7827b-b671-459a-aae9-aae432620b2e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6b170fb0f82945a198d1425d8202e66d', 'request_id': 'req-94a7827b-b671-459a-aae9-aae432620b2e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:57.875534 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-94a7827b-b671-459a-aae9-aae432620b2e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6b170fb0f82945a198d1425d8202e66d', 'request_id': 'req-94a7827b-b671-459a-aae9-aae432620b2e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:57.882201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-94a7827b-b671-459a-aae9-aae432620b2e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6b170fb0f82945a198d1425d8202e66d', 'request_id': 'req-94a7827b-b671-459a-aae9-aae432620b2e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:57.895804 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-94a7827b-b671-459a-aae9-aae432620b2e admin admin] The token cache is being invalidate because project 6b170fb0f82945a198d1425d8202e66d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:26:57.896113 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-94a7827b-b671-459a-aae9-aae432620b2e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:57.896731 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-94a7827b-b671-459a-aae9-aae432620b2e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:57.898358 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1402/2792] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:26:57 2022] DELETE /identity/v3/projects/6b170fb0f82945a198d1425d8202e66d => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:57.906118 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbefec81-f4ee-4e81-801d-17e395893f63 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:57.916014 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbefec81-f4ee-4e81-801d-17e395893f63 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:57.981329 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15edf774-c92a-4d8f-b57f-7b61650a014d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:57.981951 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15edf774-c92a-4d8f-b57f-7b61650a014d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:57.982228 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15edf774-c92a-4d8f-b57f-7b61650a014d admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:57.982497 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15edf774-c92a-4d8f-b57f-7b61650a014d admin admin] PATH_INFO: `/v3/users/56e75840a57243aa91fc24ef69c89e9d` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:57.983687 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15edf774-c92a-4d8f-b57f-7b61650a014d admin admin] RBAC: Authorizing `identity:delete_user(user_id=56e75840a57243aa91fc24ef69c89e9d)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:57.985639 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15edf774-c92a-4d8f-b57f-7b61650a014d admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:58.031126 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-15edf774-c92a-4d8f-b57f-7b61650a014d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '56e75840a57243aa91fc24ef69c89e9d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:58.031754 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-15edf774-c92a-4d8f-b57f-7b61650a014d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '56e75840a57243aa91fc24ef69c89e9d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:58.038617 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-15edf774-c92a-4d8f-b57f-7b61650a014d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '56e75840a57243aa91fc24ef69c89e9d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:58.039194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-15edf774-c92a-4d8f-b57f-7b61650a014d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '56e75840a57243aa91fc24ef69c89e9d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:58.045956 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-15edf774-c92a-4d8f-b57f-7b61650a014d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '56e75840a57243aa91fc24ef69c89e9d'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:58.063693 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1391/2793] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:26:57 2022] DELETE /identity/v3/users/56e75840a57243aa91fc24ef69c89e9d => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:58.329427 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a7827b-b671-459a-aae9-aae432620b2e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:58.330925 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-94a7827b-b671-459a-aae9-aae432620b2e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:58.344545 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a7827b-b671-459a-aae9-aae432620b2e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:58.403347 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f04fb5-2346-48dc-bd57-d81d7f053f62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:58.403897 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f04fb5-2346-48dc-bd57-d81d7f053f62 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:58.404176 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f04fb5-2346-48dc-bd57-d81d7f053f62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:58.404461 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f04fb5-2346-48dc-bd57-d81d7f053f62 admin admin] PATH_INFO: `/v3/projects/846827d35bd24fbf9ea0124a3c02eb9d` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:58.405602 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f04fb5-2346-48dc-bd57-d81d7f053f62 admin admin] RBAC: Authorizing `identity:delete_project(project_id=846827d35bd24fbf9ea0124a3c02eb9d)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:58.407691 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f04fb5-2346-48dc-bd57-d81d7f053f62 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:58.454787 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-12f04fb5-2346-48dc-bd57-d81d7f053f62 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '846827d35bd24fbf9ea0124a3c02eb9d', 'request_id': 'req-12f04fb5-2346-48dc-bd57-d81d7f053f62'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:58.461577 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-12f04fb5-2346-48dc-bd57-d81d7f053f62 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '846827d35bd24fbf9ea0124a3c02eb9d', 'request_id': 'req-12f04fb5-2346-48dc-bd57-d81d7f053f62'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:58.467776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-12f04fb5-2346-48dc-bd57-d81d7f053f62 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '846827d35bd24fbf9ea0124a3c02eb9d', 'request_id': 'req-12f04fb5-2346-48dc-bd57-d81d7f053f62'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:58.479618 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-12f04fb5-2346-48dc-bd57-d81d7f053f62 admin admin] The token cache is being invalidate because project 846827d35bd24fbf9ea0124a3c02eb9d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:26:58.479902 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-12f04fb5-2346-48dc-bd57-d81d7f053f62 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:58.480542 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-12f04fb5-2346-48dc-bd57-d81d7f053f62 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:26:58.481986 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1403/2794] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:26:58 2022] DELETE /identity/v3/projects/846827d35bd24fbf9ea0124a3c02eb9d => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:58.487853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4091c7c3-7b9a-4534-899e-1a34dd19f4ab None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:58.488312 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4091c7c3-7b9a-4534-899e-1a34dd19f4ab None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:58.488813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4091c7c3-7b9a-4534-899e-1a34dd19f4ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:58.509710 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-4091c7c3-7b9a-4534-899e-1a34dd19f4ab None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:58.510929 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4091c7c3-7b9a-4534-899e-1a34dd19f4ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:58.534907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:58.571118 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1392/2795] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:58.579031 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8f328c-b88e-444a-8c9b-04a09e1d8fe6 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:58.579314 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8f328c-b88e-444a-8c9b-04a09e1d8fe6 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:58.579583 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8f328c-b88e-444a-8c9b-04a09e1d8fe6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:58.599634 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-7f8f328c-b88e-444a-8c9b-04a09e1d8fe6 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:58.600818 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7f8f328c-b88e-444a-8c9b-04a09e1d8fe6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:58.608821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1404/2796] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:58.612972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4091c7c3-7b9a-4534-899e-1a34dd19f4ab None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:58.622712 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4091c7c3-7b9a-4534-899e-1a34dd19f4ab None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:58.627616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80309ea-cc6c-4871-969d-ced9b63a6470 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:58.628252 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80309ea-cc6c-4871-969d-ced9b63a6470 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:58.628523 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80309ea-cc6c-4871-969d-ced9b63a6470 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:58.628790 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80309ea-cc6c-4871-969d-ced9b63a6470 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:58.629929 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80309ea-cc6c-4871-969d-ced9b63a6470 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:26:58.630226 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80309ea-cc6c-4871-969d-ced9b63a6470 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:58.631870 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80309ea-cc6c-4871-969d-ced9b63a6470 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:58.647127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1393/2797] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:26:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:26:58.654410 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8f328c-b88e-444a-8c9b-04a09e1d8fe6 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:58.663803 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8f328c-b88e-444a-8c9b-04a09e1d8fe6 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:58.668376 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38626ed3-0f9d-4809-b372-23c0feff7a01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:58.668967 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38626ed3-0f9d-4809-b372-23c0feff7a01 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:58.669241 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38626ed3-0f9d-4809-b372-23c0feff7a01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:58.669508 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38626ed3-0f9d-4809-b372-23c0feff7a01 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:58.670148 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38626ed3-0f9d-4809-b372-23c0feff7a01 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:58.672074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38626ed3-0f9d-4809-b372-23c0feff7a01 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:58.698779 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1405/2798] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:26:58 2022] POST /identity/v3/projects => generated 377 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:26:58.702974 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80309ea-cc6c-4871-969d-ced9b63a6470 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:58.713377 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80309ea-cc6c-4871-969d-ced9b63a6470 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:58.768649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a491644-23ac-4bca-9562-67c599ac3d94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:58.769236 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a491644-23ac-4bca-9562-67c599ac3d94 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:58.769517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a491644-23ac-4bca-9562-67c599ac3d94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:58.769800 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a491644-23ac-4bca-9562-67c599ac3d94 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:58.770432 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a491644-23ac-4bca-9562-67c599ac3d94 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:58.772355 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a491644-23ac-4bca-9562-67c599ac3d94 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:58.792098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1394/2799] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:26:58 2022] POST /identity/v3/users => generated 289 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:26:58.796108 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38626ed3-0f9d-4809-b372-23c0feff7a01 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:58.803282 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:58.807693 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38626ed3-0f9d-4809-b372-23c0feff7a01 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:58.814560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6448b323-8a6d-4939-ab01-3c2f6c74e08c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:58.815586 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6448b323-8a6d-4939-ab01-3c2f6c74e08c admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:58.815973 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6448b323-8a6d-4939-ab01-3c2f6c74e08c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:58.816429 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6448b323-8a6d-4939-ab01-3c2f6c74e08c admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:58.817706 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6448b323-8a6d-4939-ab01-3c2f6c74e08c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:58.819447 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6448b323-8a6d-4939-ab01-3c2f6c74e08c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:58.829592 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1406/2800] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:58 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:26:58.833553 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a491644-23ac-4bca-9562-67c599ac3d94 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:58.843510 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a491644-23ac-4bca-9562-67c599ac3d94 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:58.848064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6223cfdc-9873-4f91-8e51-d233150e7b4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:58.848659 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6223cfdc-9873-4f91-8e51-d233150e7b4e admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:58.848935 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6223cfdc-9873-4f91-8e51-d233150e7b4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:58.849202 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6223cfdc-9873-4f91-8e51-d233150e7b4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:58.849825 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6223cfdc-9873-4f91-8e51-d233150e7b4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:58.851452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6223cfdc-9873-4f91-8e51-d233150e7b4e admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:58.861788 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1395/2801] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:58 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:26:58.866261 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6448b323-8a6d-4939-ab01-3c2f6c74e08c admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:58.876307 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6448b323-8a6d-4939-ab01-3c2f6c74e08c admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:58.881512 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d40094-6d05-405f-bcdb-ef6561584dcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:58.882167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d40094-6d05-405f-bcdb-ef6561584dcc admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:58.882447 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d40094-6d05-405f-bcdb-ef6561584dcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:58.882723 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d40094-6d05-405f-bcdb-ef6561584dcc admin admin] PATH_INFO: `/v3/projects/387df9fde8e04fd88e6f9bfad89c4604/users/e6056360a874436eb12e0b9336dcd79a/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:58.897208 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d40094-6d05-405f-bcdb-ef6561584dcc admin admin] RBAC: Authorizing `identity:create_grant(project_id=387df9fde8e04fd88e6f9bfad89c4604, user_id=e6056360a874436eb12e0b9336dcd79a, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:58.899920 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d40094-6d05-405f-bcdb-ef6561584dcc admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:58.912600 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1407/2802] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:26:58 2022] PUT /identity/v3/projects/387df9fde8e04fd88e6f9bfad89c4604/users/e6056360a874436eb12e0b9336dcd79a/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:58.922732 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ed4b23-898b-4636-9653-38253e07f7f2 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:58.923054 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ed4b23-898b-4636-9653-38253e07f7f2 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:58.923331 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ed4b23-898b-4636-9653-38253e07f7f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:58.956634 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-44ed4b23-898b-4636-9653-38253e07f7f2 None None] MFA Rules not processed for user `e6056360a874436eb12e0b9336dcd79a`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:58.958048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-44ed4b23-898b-4636-9653-38253e07f7f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:59.031484 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1396/2803] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:58 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:59.035969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d40094-6d05-405f-bcdb-ef6561584dcc admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:59.096707 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d40094-6d05-405f-bcdb-ef6561584dcc admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:59.101525 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d19135-b117-48cd-8e37-d4faa8c7cd86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:59.102120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d19135-b117-48cd-8e37-d4faa8c7cd86 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:59.102395 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d19135-b117-48cd-8e37-d4faa8c7cd86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:59.102690 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d19135-b117-48cd-8e37-d4faa8c7cd86 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:59.103359 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d19135-b117-48cd-8e37-d4faa8c7cd86 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:59.105257 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d19135-b117-48cd-8e37-d4faa8c7cd86 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:59.131817 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1408/2804] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:26:59 2022] POST /identity/v3/projects => generated 379 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:26:59.135880 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ed4b23-898b-4636-9653-38253e07f7f2 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:59.146486 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ed4b23-898b-4636-9653-38253e07f7f2 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:59.166159 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:59.203531 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f72acb-41dc-4933-abdb-3221ebce1cd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:59.204159 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f72acb-41dc-4933-abdb-3221ebce1cd3 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:59.204439 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f72acb-41dc-4933-abdb-3221ebce1cd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:59.204712 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f72acb-41dc-4933-abdb-3221ebce1cd3 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:59.205330 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f72acb-41dc-4933-abdb-3221ebce1cd3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:59.207284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f72acb-41dc-4933-abdb-3221ebce1cd3 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:59.228240 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1397/2805] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:26:59 2022] POST /identity/v3/users => generated 290 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:26:59.232257 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d19135-b117-48cd-8e37-d4faa8c7cd86 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:59.243026 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d19135-b117-48cd-8e37-d4faa8c7cd86 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:59.247553 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605528dc-64d8-44d4-84b0-894c6372952f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:59.248143 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605528dc-64d8-44d4-84b0-894c6372952f admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:59.248416 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605528dc-64d8-44d4-84b0-894c6372952f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:59.248684 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605528dc-64d8-44d4-84b0-894c6372952f admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:59.249312 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605528dc-64d8-44d4-84b0-894c6372952f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:59.250955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605528dc-64d8-44d4-84b0-894c6372952f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:59.262013 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1409/2806] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:59 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:26:59.266249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f72acb-41dc-4933-abdb-3221ebce1cd3 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:59.280236 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f72acb-41dc-4933-abdb-3221ebce1cd3 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:59.285223 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54da0de6-1bb0-46dc-94fd-ebe5f7be1ed3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:59.286344 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54da0de6-1bb0-46dc-94fd-ebe5f7be1ed3 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:59.286619 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54da0de6-1bb0-46dc-94fd-ebe5f7be1ed3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:59.286906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54da0de6-1bb0-46dc-94fd-ebe5f7be1ed3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:59.288646 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54da0de6-1bb0-46dc-94fd-ebe5f7be1ed3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:59.289336 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54da0de6-1bb0-46dc-94fd-ebe5f7be1ed3 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:59.299916 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1398/2807] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:59 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:26:59.304902 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605528dc-64d8-44d4-84b0-894c6372952f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:59.314930 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605528dc-64d8-44d4-84b0-894c6372952f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:59.319536 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8cc2fb-d0e1-480e-8dd7-784cd7098a9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:59.320175 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8cc2fb-d0e1-480e-8dd7-784cd7098a9e admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:59.320457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8cc2fb-d0e1-480e-8dd7-784cd7098a9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:59.320732 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8cc2fb-d0e1-480e-8dd7-784cd7098a9e admin admin] PATH_INFO: `/v3/projects/2f638b7fa844420ebdc9672ad201386f/users/9639eccf91774c55ba673d3c0eb32aef/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:59.334997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8cc2fb-d0e1-480e-8dd7-784cd7098a9e admin admin] RBAC: Authorizing `identity:create_grant(project_id=2f638b7fa844420ebdc9672ad201386f, user_id=9639eccf91774c55ba673d3c0eb32aef, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:59.337655 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8cc2fb-d0e1-480e-8dd7-784cd7098a9e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:59.350989 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1410/2808] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:26:59 2022] PUT /identity/v3/projects/2f638b7fa844420ebdc9672ad201386f/users/9639eccf91774c55ba673d3c0eb32aef/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:59.362169 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c19dc3d-47ed-40f8-bcb8-06d4b39eced2 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:59.362558 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c19dc3d-47ed-40f8-bcb8-06d4b39eced2 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:59.362836 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c19dc3d-47ed-40f8-bcb8-06d4b39eced2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:59.390438 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-8c19dc3d-47ed-40f8-bcb8-06d4b39eced2 None None] MFA Rules not processed for user `9639eccf91774c55ba673d3c0eb32aef`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:59.391661 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-8c19dc3d-47ed-40f8-bcb8-06d4b39eced2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:59.455558 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1399/2809] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:59 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:26:59.459929 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8cc2fb-d0e1-480e-8dd7-784cd7098a9e admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:59.520438 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8cc2fb-d0e1-480e-8dd7-784cd7098a9e admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:59.524981 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba9b606-b974-47af-af0f-26d7269f6297 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:59.525810 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba9b606-b974-47af-af0f-26d7269f6297 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:59.526087 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba9b606-b974-47af-af0f-26d7269f6297 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:59.526359 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba9b606-b974-47af-af0f-26d7269f6297 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:59.527011 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba9b606-b974-47af-af0f-26d7269f6297 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:59.528890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba9b606-b974-47af-af0f-26d7269f6297 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:59.555700 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1411/2810] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:26:59 2022] POST /identity/v3/projects => generated 377 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:26:59.560161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c19dc3d-47ed-40f8-bcb8-06d4b39eced2 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:59.571303 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c19dc3d-47ed-40f8-bcb8-06d4b39eced2 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:59.628476 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f083d554-3fb3-4feb-917b-a4139452dbb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:59.629084 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f083d554-3fb3-4feb-917b-a4139452dbb7 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:59.629364 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f083d554-3fb3-4feb-917b-a4139452dbb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:59.629645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f083d554-3fb3-4feb-917b-a4139452dbb7 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:59.630261 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f083d554-3fb3-4feb-917b-a4139452dbb7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:59.632209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f083d554-3fb3-4feb-917b-a4139452dbb7 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:59.653380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1400/2811] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:26:59 2022] POST /identity/v3/users => generated 295 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:26:59.657393 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba9b606-b974-47af-af0f-26d7269f6297 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:59.664691 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:59.668753 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba9b606-b974-47af-af0f-26d7269f6297 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:59.674201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed14919-70f8-4645-b32c-4e3f5a7a10a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:59.674799 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed14919-70f8-4645-b32c-4e3f5a7a10a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:59.675108 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed14919-70f8-4645-b32c-4e3f5a7a10a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:59.675382 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed14919-70f8-4645-b32c-4e3f5a7a10a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:59.676016 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed14919-70f8-4645-b32c-4e3f5a7a10a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:59.677635 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed14919-70f8-4645-b32c-4e3f5a7a10a5 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:59.688303 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1412/2812] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:59 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:26:59.692357 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f083d554-3fb3-4feb-917b-a4139452dbb7 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:59.702098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f083d554-3fb3-4feb-917b-a4139452dbb7 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:59.706718 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6f9759-7b20-4b26-9397-a5b8d26c9179 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:59.707899 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6f9759-7b20-4b26-9397-a5b8d26c9179 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:59.708159 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6f9759-7b20-4b26-9397-a5b8d26c9179 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:59.708429 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6f9759-7b20-4b26-9397-a5b8d26c9179 admin admin] PATH_INFO: `/v3/projects/cbdb14e107014cac8fb9652487eda841/users/915b0f2f1dd247c880ee14fd5ac63169/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:59.723028 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6f9759-7b20-4b26-9397-a5b8d26c9179 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cbdb14e107014cac8fb9652487eda841, user_id=915b0f2f1dd247c880ee14fd5ac63169, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:59.725783 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6f9759-7b20-4b26-9397-a5b8d26c9179 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:59.733684 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:26:59.740202 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1401/2813] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:26:59 2022] PUT /identity/v3/projects/cbdb14e107014cac8fb9652487eda841/users/915b0f2f1dd247c880ee14fd5ac63169/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:59.744156 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed14919-70f8-4645-b32c-4e3f5a7a10a5 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:59.754028 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed14919-70f8-4645-b32c-4e3f5a7a10a5 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:59.810212 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab255e80-2696-49b1-8124-96e910a34f3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:59.810878 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab255e80-2696-49b1-8124-96e910a34f3c admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:59.811164 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab255e80-2696-49b1-8124-96e910a34f3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:59.811441 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab255e80-2696-49b1-8124-96e910a34f3c admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:59.812076 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab255e80-2696-49b1-8124-96e910a34f3c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:59.813672 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab255e80-2696-49b1-8124-96e910a34f3c admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:59.824521 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1413/2814] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:26:59 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:26:59.828542 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6f9759-7b20-4b26-9397-a5b8d26c9179 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:59.839201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6f9759-7b20-4b26-9397-a5b8d26c9179 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:26:59.843915 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0981de-cf97-4d8f-8016-fc9b7e962e59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:26:59.844567 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0981de-cf97-4d8f-8016-fc9b7e962e59 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:59.844847 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0981de-cf97-4d8f-8016-fc9b7e962e59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:59.845119 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0981de-cf97-4d8f-8016-fc9b7e962e59 admin admin] PATH_INFO: `/v3/projects/cbdb14e107014cac8fb9652487eda841/users/915b0f2f1dd247c880ee14fd5ac63169/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:59.847041 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0981de-cf97-4d8f-8016-fc9b7e962e59 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cbdb14e107014cac8fb9652487eda841, user_id=915b0f2f1dd247c880ee14fd5ac63169, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:26:59.849844 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0981de-cf97-4d8f-8016-fc9b7e962e59 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:26:59.860090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1402/2815] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:26:59 2022] PUT /identity/v3/projects/cbdb14e107014cac8fb9652487eda841/users/915b0f2f1dd247c880ee14fd5ac63169/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:26:59.870903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2716a902-3e9a-43b2-a30e-e952d9070cc7 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:26:59.871191 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2716a902-3e9a-43b2-a30e-e952d9070cc7 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:26:59.871461 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2716a902-3e9a-43b2-a30e-e952d9070cc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:26:59.900796 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-2716a902-3e9a-43b2-a30e-e952d9070cc7 None None] MFA Rules not processed for user `915b0f2f1dd247c880ee14fd5ac63169`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:26:59.902097 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2716a902-3e9a-43b2-a30e-e952d9070cc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:26:59.971957 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1414/2816] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:26:59 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:26:59.984065 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0981de-cf97-4d8f-8016-fc9b7e962e59 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:26:59.985848 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0b0981de-cf97-4d8f-8016-fc9b7e962e59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:00.044167 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0981de-cf97-4d8f-8016-fc9b7e962e59 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:00.047828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfe5666-26fb-4039-b2df-78fd72b1a506 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:00.048433 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfe5666-26fb-4039-b2df-78fd72b1a506 service cinder] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:00.048713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfe5666-26fb-4039-b2df-78fd72b1a506 service cinder] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:00.048983 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfe5666-26fb-4039-b2df-78fd72b1a506 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:00.058788 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfe5666-26fb-4039-b2df-78fd72b1a506 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:00.060241 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfe5666-26fb-4039-b2df-78fd72b1a506 service cinder] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:00.069589 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1403/2817] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:26:59 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:27:00.109700 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2716a902-3e9a-43b2-a30e-e952d9070cc7 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:00.111183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2716a902-3e9a-43b2-a30e-e952d9070cc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:00.123587 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2716a902-3e9a-43b2-a30e-e952d9070cc7 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:00.127133 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b22fcc9-2422-4285-a47f-60a0bae1d46f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:00.127722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b22fcc9-2422-4285-a47f-60a0bae1d46f service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:00.128010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b22fcc9-2422-4285-a47f-60a0bae1d46f service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:00.128262 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b22fcc9-2422-4285-a47f-60a0bae1d46f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:00.185471 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b22fcc9-2422-4285-a47f-60a0bae1d46f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:00.186906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b22fcc9-2422-4285-a47f-60a0bae1d46f service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:00.195650 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1415/2818] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:27:00 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:27:03.049712 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfe5666-26fb-4039-b2df-78fd72b1a506 service cinder] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:03.113791 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfe5666-26fb-4039-b2df-78fd72b1a506 service cinder] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:03.118682 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e19e49-62c3-43a9-a13e-dbb27dd32da9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:03.119378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e19e49-62c3-43a9-a13e-dbb27dd32da9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:03.119662 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e19e49-62c3-43a9-a13e-dbb27dd32da9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:03.119935 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e19e49-62c3-43a9-a13e-dbb27dd32da9 admin admin] PATH_INFO: `/v3/users/e6056360a874436eb12e0b9336dcd79a` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:03.121057 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e19e49-62c3-43a9-a13e-dbb27dd32da9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e6056360a874436eb12e0b9336dcd79a)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:03.123293 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e19e49-62c3-43a9-a13e-dbb27dd32da9 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:03.169680 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-20e19e49-62c3-43a9-a13e-dbb27dd32da9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e6056360a874436eb12e0b9336dcd79a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:03.170337 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-20e19e49-62c3-43a9-a13e-dbb27dd32da9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e6056360a874436eb12e0b9336dcd79a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:03.176737 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-20e19e49-62c3-43a9-a13e-dbb27dd32da9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e6056360a874436eb12e0b9336dcd79a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:03.177425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-20e19e49-62c3-43a9-a13e-dbb27dd32da9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e6056360a874436eb12e0b9336dcd79a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:03.184949 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-20e19e49-62c3-43a9-a13e-dbb27dd32da9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e6056360a874436eb12e0b9336dcd79a'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:03.202402 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1404/2819] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:27:03 2022] DELETE /identity/v3/users/e6056360a874436eb12e0b9336dcd79a => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:27:03.206927 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5ca701-ec8c-4111-be0d-271411521c67 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:03.207408 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5ca701-ec8c-4111-be0d-271411521c67 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:03.207515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5ca701-ec8c-4111-be0d-271411521c67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:03.226110 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ad5ca701-ec8c-4111-be0d-271411521c67 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:03.227340 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ad5ca701-ec8c-4111-be0d-271411521c67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:03.235570 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:03.286180 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1416/2820] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:03.295879 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e19e49-62c3-43a9-a13e-dbb27dd32da9 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:03.297276 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-20e19e49-62c3-43a9-a13e-dbb27dd32da9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:03.307589 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:03.355876 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e19e49-62c3-43a9-a13e-dbb27dd32da9 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:03.359512 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d34ad5-b4b2-48bb-8ab5-bec3be9c3dac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:03.360112 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d34ad5-b4b2-48bb-8ab5-bec3be9c3dac service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:03.360389 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d34ad5-b4b2-48bb-8ab5-bec3be9c3dac service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:03.360655 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d34ad5-b4b2-48bb-8ab5-bec3be9c3dac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:03.370873 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d34ad5-b4b2-48bb-8ab5-bec3be9c3dac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:03.372121 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d34ad5-b4b2-48bb-8ab5-bec3be9c3dac service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:03.384247 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1405/2821] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:27:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:27:03.659906 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5ca701-ec8c-4111-be0d-271411521c67 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:03.661477 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ad5ca701-ec8c-4111-be0d-271411521c67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:03.674507 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5ca701-ec8c-4111-be0d-271411521c67 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:03.679488 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20c2344-c277-4770-94b9-78841a25bc43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:03.680636 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20c2344-c277-4770-94b9-78841a25bc43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:03.681043 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20c2344-c277-4770-94b9-78841a25bc43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:03.681316 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20c2344-c277-4770-94b9-78841a25bc43 admin admin] PATH_INFO: `/v3/projects/387df9fde8e04fd88e6f9bfad89c4604` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:03.682434 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20c2344-c277-4770-94b9-78841a25bc43 admin admin] RBAC: Authorizing `identity:delete_project(project_id=387df9fde8e04fd88e6f9bfad89c4604)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:03.684359 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20c2344-c277-4770-94b9-78841a25bc43 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:03.731818 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c20c2344-c277-4770-94b9-78841a25bc43 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '387df9fde8e04fd88e6f9bfad89c4604', 'request_id': 'req-c20c2344-c277-4770-94b9-78841a25bc43'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:03.737289 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c20c2344-c277-4770-94b9-78841a25bc43 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '387df9fde8e04fd88e6f9bfad89c4604', 'request_id': 'req-c20c2344-c277-4770-94b9-78841a25bc43'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:03.744225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c20c2344-c277-4770-94b9-78841a25bc43 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '387df9fde8e04fd88e6f9bfad89c4604', 'request_id': 'req-c20c2344-c277-4770-94b9-78841a25bc43'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:03.756002 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c20c2344-c277-4770-94b9-78841a25bc43 admin admin] The token cache is being invalidate because project 387df9fde8e04fd88e6f9bfad89c4604 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:27:03.756289 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c20c2344-c277-4770-94b9-78841a25bc43 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:03.756900 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-c20c2344-c277-4770-94b9-78841a25bc43 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:03.758374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1417/2822] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:27:03 2022] DELETE /identity/v3/projects/387df9fde8e04fd88e6f9bfad89c4604 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:27:03.762370 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d34ad5-b4b2-48bb-8ab5-bec3be9c3dac service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:03.772106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d34ad5-b4b2-48bb-8ab5-bec3be9c3dac service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:03.829968 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a300e274-21a4-4dc4-ae0a-45309d1fd2d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:03.830596 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a300e274-21a4-4dc4-ae0a-45309d1fd2d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:03.830898 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a300e274-21a4-4dc4-ae0a-45309d1fd2d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:03.831178 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a300e274-21a4-4dc4-ae0a-45309d1fd2d5 admin admin] PATH_INFO: `/v3/users/9639eccf91774c55ba673d3c0eb32aef` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:03.832296 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a300e274-21a4-4dc4-ae0a-45309d1fd2d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9639eccf91774c55ba673d3c0eb32aef)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:03.834204 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a300e274-21a4-4dc4-ae0a-45309d1fd2d5 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:03.878709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a300e274-21a4-4dc4-ae0a-45309d1fd2d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9639eccf91774c55ba673d3c0eb32aef'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:03.879381 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a300e274-21a4-4dc4-ae0a-45309d1fd2d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9639eccf91774c55ba673d3c0eb32aef'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:03.886570 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a300e274-21a4-4dc4-ae0a-45309d1fd2d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9639eccf91774c55ba673d3c0eb32aef'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:03.887202 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a300e274-21a4-4dc4-ae0a-45309d1fd2d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9639eccf91774c55ba673d3c0eb32aef'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:03.894140 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-a300e274-21a4-4dc4-ae0a-45309d1fd2d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9639eccf91774c55ba673d3c0eb32aef'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:03.912015 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1406/2823] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:27:03 2022] DELETE /identity/v3/users/9639eccf91774c55ba673d3c0eb32aef => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:27:04.175313 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20c2344-c277-4770-94b9-78841a25bc43 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:04.176742 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c20c2344-c277-4770-94b9-78841a25bc43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:04.190060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20c2344-c277-4770-94b9-78841a25bc43 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:04.251065 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bee57b-b57d-42ab-93e6-b2a996fb1763 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:04.251801 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bee57b-b57d-42ab-93e6-b2a996fb1763 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:04.252164 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bee57b-b57d-42ab-93e6-b2a996fb1763 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:04.252437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bee57b-b57d-42ab-93e6-b2a996fb1763 admin admin] PATH_INFO: `/v3/projects/2f638b7fa844420ebdc9672ad201386f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:04.253634 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bee57b-b57d-42ab-93e6-b2a996fb1763 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2f638b7fa844420ebdc9672ad201386f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:04.261130 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bee57b-b57d-42ab-93e6-b2a996fb1763 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:04.295096 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:04.303171 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f1bee57b-b57d-42ab-93e6-b2a996fb1763 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2f638b7fa844420ebdc9672ad201386f', 'request_id': 'req-f1bee57b-b57d-42ab-93e6-b2a996fb1763'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:04.308847 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f1bee57b-b57d-42ab-93e6-b2a996fb1763 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2f638b7fa844420ebdc9672ad201386f', 'request_id': 'req-f1bee57b-b57d-42ab-93e6-b2a996fb1763'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:04.314819 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f1bee57b-b57d-42ab-93e6-b2a996fb1763 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2f638b7fa844420ebdc9672ad201386f', 'request_id': 'req-f1bee57b-b57d-42ab-93e6-b2a996fb1763'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:04.327424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f1bee57b-b57d-42ab-93e6-b2a996fb1763 admin admin] The token cache is being invalidate because project 2f638b7fa844420ebdc9672ad201386f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:27:04.327717 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f1bee57b-b57d-42ab-93e6-b2a996fb1763 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:04.328348 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f1bee57b-b57d-42ab-93e6-b2a996fb1763 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:04.329825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1418/2824] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:27:04 2022] DELETE /identity/v3/projects/2f638b7fa844420ebdc9672ad201386f => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:27:04.333802 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a300e274-21a4-4dc4-ae0a-45309d1fd2d5 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:04.335218 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-a300e274-21a4-4dc4-ae0a-45309d1fd2d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:04.348192 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a300e274-21a4-4dc4-ae0a-45309d1fd2d5 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:04.373167 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:04.405872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba60c70-a02b-4a18-ae40-1264ed02b046 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:04.406504 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba60c70-a02b-4a18-ae40-1264ed02b046 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:04.406781 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba60c70-a02b-4a18-ae40-1264ed02b046 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:04.407082 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba60c70-a02b-4a18-ae40-1264ed02b046 admin admin] PATH_INFO: `/v3/users/915b0f2f1dd247c880ee14fd5ac63169` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:04.410416 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba60c70-a02b-4a18-ae40-1264ed02b046 admin admin] RBAC: Authorizing `identity:delete_user(user_id=915b0f2f1dd247c880ee14fd5ac63169)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:04.412346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba60c70-a02b-4a18-ae40-1264ed02b046 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:04.457376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7ba60c70-a02b-4a18-ae40-1264ed02b046 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '915b0f2f1dd247c880ee14fd5ac63169'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:04.458030 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7ba60c70-a02b-4a18-ae40-1264ed02b046 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '915b0f2f1dd247c880ee14fd5ac63169'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:04.465853 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7ba60c70-a02b-4a18-ae40-1264ed02b046 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '915b0f2f1dd247c880ee14fd5ac63169'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:04.466632 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7ba60c70-a02b-4a18-ae40-1264ed02b046 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '915b0f2f1dd247c880ee14fd5ac63169'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:04.473528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-7ba60c70-a02b-4a18-ae40-1264ed02b046 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '915b0f2f1dd247c880ee14fd5ac63169'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:04.490104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1407/2825] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:27:04 2022] DELETE /identity/v3/users/915b0f2f1dd247c880ee14fd5ac63169 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:27:04.789368 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bee57b-b57d-42ab-93e6-b2a996fb1763 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:04.791018 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f1bee57b-b57d-42ab-93e6-b2a996fb1763 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:04.818660 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bee57b-b57d-42ab-93e6-b2a996fb1763 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:04.880995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9340fc2-d984-4243-bcab-79acac352f8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:04.881631 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9340fc2-d984-4243-bcab-79acac352f8f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:04.881911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9340fc2-d984-4243-bcab-79acac352f8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:04.882183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9340fc2-d984-4243-bcab-79acac352f8f admin admin] PATH_INFO: `/v3/projects/cbdb14e107014cac8fb9652487eda841` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:04.883417 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9340fc2-d984-4243-bcab-79acac352f8f admin admin] RBAC: Authorizing `identity:delete_project(project_id=cbdb14e107014cac8fb9652487eda841)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:04.885302 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9340fc2-d984-4243-bcab-79acac352f8f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:04.930994 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b9340fc2-d984-4243-bcab-79acac352f8f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cbdb14e107014cac8fb9652487eda841', 'request_id': 'req-b9340fc2-d984-4243-bcab-79acac352f8f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:04.936421 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b9340fc2-d984-4243-bcab-79acac352f8f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cbdb14e107014cac8fb9652487eda841', 'request_id': 'req-b9340fc2-d984-4243-bcab-79acac352f8f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:04.942389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b9340fc2-d984-4243-bcab-79acac352f8f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cbdb14e107014cac8fb9652487eda841', 'request_id': 'req-b9340fc2-d984-4243-bcab-79acac352f8f'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:04.955221 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b9340fc2-d984-4243-bcab-79acac352f8f admin admin] The token cache is being invalidate because project cbdb14e107014cac8fb9652487eda841 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:27:04.955512 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b9340fc2-d984-4243-bcab-79acac352f8f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:04.956151 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b9340fc2-d984-4243-bcab-79acac352f8f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:04.957618 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1419/2826] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:27:04 2022] DELETE /identity/v3/projects/cbdb14e107014cac8fb9652487eda841 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:27:04.965066 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d219f30d-9175-4e9c-8f7c-7ca7bab795b6 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:04.965458 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d219f30d-9175-4e9c-8f7c-7ca7bab795b6 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:04.965746 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d219f30d-9175-4e9c-8f7c-7ca7bab795b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:04.985806 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-d219f30d-9175-4e9c-8f7c-7ca7bab795b6 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:04.987021 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-d219f30d-9175-4e9c-8f7c-7ca7bab795b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:05.043335 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1408/2827] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:05.052829 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11927053-8f73-4a9a-ba60-18b90b1991d0 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:05.053121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11927053-8f73-4a9a-ba60-18b90b1991d0 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:05.053653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11927053-8f73-4a9a-ba60-18b90b1991d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:05.073513 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-11927053-8f73-4a9a-ba60-18b90b1991d0 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:05.074725 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-11927053-8f73-4a9a-ba60-18b90b1991d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:05.082551 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1420/2828] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:05.087307 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d219f30d-9175-4e9c-8f7c-7ca7bab795b6 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:05.096994 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d219f30d-9175-4e9c-8f7c-7ca7bab795b6 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:05.101514 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e542be-4359-469e-9345-626751cc8d80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:05.102104 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e542be-4359-469e-9345-626751cc8d80 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:05.102381 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e542be-4359-469e-9345-626751cc8d80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:05.102654 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e542be-4359-469e-9345-626751cc8d80 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:05.103328 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e542be-4359-469e-9345-626751cc8d80 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:27:05.103626 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e542be-4359-469e-9345-626751cc8d80 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:05.105215 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e542be-4359-469e-9345-626751cc8d80 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:05.116566 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1409/2829] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:27:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:27:05.120752 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11927053-8f73-4a9a-ba60-18b90b1991d0 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:05.129935 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11927053-8f73-4a9a-ba60-18b90b1991d0 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:05.135104 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f66e6c8-42a9-4cd1-8dbf-3a79a574aacb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:05.135695 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f66e6c8-42a9-4cd1-8dbf-3a79a574aacb admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:05.135973 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f66e6c8-42a9-4cd1-8dbf-3a79a574aacb admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:05.136244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f66e6c8-42a9-4cd1-8dbf-3a79a574aacb admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:05.136874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f66e6c8-42a9-4cd1-8dbf-3a79a574aacb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:05.138745 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f66e6c8-42a9-4cd1-8dbf-3a79a574aacb admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:05.165463 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1421/2830] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:27:05 2022] POST /identity/v3/projects => generated 385 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:27:05.169670 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e542be-4359-469e-9345-626751cc8d80 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:05.179737 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e542be-4359-469e-9345-626751cc8d80 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:05.211868 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:05.237379 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1fb34e-2a1b-4db1-b66a-5e3c00d33f5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:05.237980 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1fb34e-2a1b-4db1-b66a-5e3c00d33f5b admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:05.238259 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1fb34e-2a1b-4db1-b66a-5e3c00d33f5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:05.238529 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1fb34e-2a1b-4db1-b66a-5e3c00d33f5b admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:05.239197 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1fb34e-2a1b-4db1-b66a-5e3c00d33f5b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:05.241100 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1fb34e-2a1b-4db1-b66a-5e3c00d33f5b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:05.262212 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1410/2831] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:27:05 2022] POST /identity/v3/users => generated 293 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:27:05.266365 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f66e6c8-42a9-4cd1-8dbf-3a79a574aacb admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:05.277377 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f66e6c8-42a9-4cd1-8dbf-3a79a574aacb admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:05.281964 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffb47fa-d6f9-442f-9f9d-a8419ce19095 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:05.282559 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffb47fa-d6f9-442f-9f9d-a8419ce19095 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:05.282873 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffb47fa-d6f9-442f-9f9d-a8419ce19095 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:05.283138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffb47fa-d6f9-442f-9f9d-a8419ce19095 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:05.283768 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffb47fa-d6f9-442f-9f9d-a8419ce19095 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:05.285336 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffb47fa-d6f9-442f-9f9d-a8419ce19095 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:05.295148 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1422/2832] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:27:05 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:27:05.298828 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1fb34e-2a1b-4db1-b66a-5e3c00d33f5b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:05.308783 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1fb34e-2a1b-4db1-b66a-5e3c00d33f5b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:05.313309 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423422d6-ddbf-43d3-b502-093780b0082a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:05.313887 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423422d6-ddbf-43d3-b502-093780b0082a admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:05.314162 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423422d6-ddbf-43d3-b502-093780b0082a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:05.314427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423422d6-ddbf-43d3-b502-093780b0082a admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:05.315080 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423422d6-ddbf-43d3-b502-093780b0082a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:05.316635 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423422d6-ddbf-43d3-b502-093780b0082a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:05.326793 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1411/2833] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:27:05 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:27:05.330661 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffb47fa-d6f9-442f-9f9d-a8419ce19095 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:05.341114 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffb47fa-d6f9-442f-9f9d-a8419ce19095 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:05.345696 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea938b8-3ecb-47a1-b847-d36c7c8e7c7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:05.346332 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea938b8-3ecb-47a1-b847-d36c7c8e7c7a admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:05.346612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea938b8-3ecb-47a1-b847-d36c7c8e7c7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:05.346914 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea938b8-3ecb-47a1-b847-d36c7c8e7c7a admin admin] PATH_INFO: `/v3/projects/3d8d0a19f1934a49aa1e0fb7ccfaa5aa/users/0b18be1a47054a20a8de8d0343daefff/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:05.362183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea938b8-3ecb-47a1-b847-d36c7c8e7c7a admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d8d0a19f1934a49aa1e0fb7ccfaa5aa, user_id=0b18be1a47054a20a8de8d0343daefff, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:05.364821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea938b8-3ecb-47a1-b847-d36c7c8e7c7a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:05.367043 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:05.379653 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1423/2834] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:27:05 2022] PUT /identity/v3/projects/3d8d0a19f1934a49aa1e0fb7ccfaa5aa/users/0b18be1a47054a20a8de8d0343daefff/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:27:05.446429 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e45302d-b247-478f-be43-96082022675d None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:05.447046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e45302d-b247-478f-be43-96082022675d None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:05.447046 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e45302d-b247-478f-be43-96082022675d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:05.474980 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-1e45302d-b247-478f-be43-96082022675d None None] MFA Rules not processed for user `0b18be1a47054a20a8de8d0343daefff`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:05.476216 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1e45302d-b247-478f-be43-96082022675d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:05.538820 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1412/2835] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:05 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:05.549869 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea938b8-3ecb-47a1-b847-d36c7c8e7c7a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:05.551518 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-2ea938b8-3ecb-47a1-b847-d36c7c8e7c7a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:05.628955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea938b8-3ecb-47a1-b847-d36c7c8e7c7a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:05.633282 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841627a2-89a6-4b88-a913-207c1871b92d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:05.634010 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841627a2-89a6-4b88-a913-207c1871b92d service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:05.634328 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841627a2-89a6-4b88-a913-207c1871b92d service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:05.634605 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841627a2-89a6-4b88-a913-207c1871b92d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:05.643045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841627a2-89a6-4b88-a913-207c1871b92d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:05.644473 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841627a2-89a6-4b88-a913-207c1871b92d service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:05.653407 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1424/2836] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:27:05 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:27:05.671637 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e45302d-b247-478f-be43-96082022675d None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:05.673130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1e45302d-b247-478f-be43-96082022675d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:05.741511 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e45302d-b247-478f-be43-96082022675d None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:05.746594 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dff8d5c-837d-4e3d-87ff-78c1e90023d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:05.747257 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dff8d5c-837d-4e3d-87ff-78c1e90023d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:05.747540 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dff8d5c-837d-4e3d-87ff-78c1e90023d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:05.747814 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dff8d5c-837d-4e3d-87ff-78c1e90023d2 admin admin] PATH_INFO: `/v3/users/82e4e65638e54903acd34afa87f47505` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:05.748945 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dff8d5c-837d-4e3d-87ff-78c1e90023d2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=82e4e65638e54903acd34afa87f47505)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:05.750877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dff8d5c-837d-4e3d-87ff-78c1e90023d2 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:05.798873 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:05.800689 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3dff8d5c-837d-4e3d-87ff-78c1e90023d2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '82e4e65638e54903acd34afa87f47505'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:05.801383 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3dff8d5c-837d-4e3d-87ff-78c1e90023d2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '82e4e65638e54903acd34afa87f47505'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:05.811106 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3dff8d5c-837d-4e3d-87ff-78c1e90023d2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '82e4e65638e54903acd34afa87f47505'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:05.812403 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3dff8d5c-837d-4e3d-87ff-78c1e90023d2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '82e4e65638e54903acd34afa87f47505'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:05.820883 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-3dff8d5c-837d-4e3d-87ff-78c1e90023d2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '82e4e65638e54903acd34afa87f47505'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:05.839934 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1413/2837] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:27:05 2022] DELETE /identity/v3/users/82e4e65638e54903acd34afa87f47505 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:27:05.845064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858fe3ef-df77-45a0-882c-24e88219e85f None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:05.845064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858fe3ef-df77-45a0-882c-24e88219e85f None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:05.845440 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858fe3ef-df77-45a0-882c-24e88219e85f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:05.866321 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-858fe3ef-df77-45a0-882c-24e88219e85f None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:05.868256 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-858fe3ef-df77-45a0-882c-24e88219e85f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:05.937068 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1425/2838] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:05.958877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dff8d5c-837d-4e3d-87ff-78c1e90023d2 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:05.958877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-3dff8d5c-837d-4e3d-87ff-78c1e90023d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:06.010501 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dff8d5c-837d-4e3d-87ff-78c1e90023d2 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:06.014162 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28388c0d-d215-4b0e-bf19-4ea4c3bb4d0f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:06.014769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28388c0d-d215-4b0e-bf19-4ea4c3bb4d0f service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:06.015176 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28388c0d-d215-4b0e-bf19-4ea4c3bb4d0f service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:06.015439 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28388c0d-d215-4b0e-bf19-4ea4c3bb4d0f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:06.024421 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28388c0d-d215-4b0e-bf19-4ea4c3bb4d0f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:06.025903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28388c0d-d215-4b0e-bf19-4ea4c3bb4d0f service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:06.035311 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1414/2839] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:27:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:27:06.301068 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858fe3ef-df77-45a0-882c-24e88219e85f None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:06.302510 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-858fe3ef-df77-45a0-882c-24e88219e85f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:06.315596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858fe3ef-df77-45a0-882c-24e88219e85f None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:06.320340 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b45d657-acc2-426b-8085-8fffb74a853e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:06.320960 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b45d657-acc2-426b-8085-8fffb74a853e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:06.321722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b45d657-acc2-426b-8085-8fffb74a853e admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:06.321995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b45d657-acc2-426b-8085-8fffb74a853e admin admin] PATH_INFO: `/v3/projects/9fca31eaad0a453b9ee50c73fd5b99a5` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:06.323383 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b45d657-acc2-426b-8085-8fffb74a853e admin admin] RBAC: Authorizing `identity:delete_project(project_id=9fca31eaad0a453b9ee50c73fd5b99a5)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:06.325299 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b45d657-acc2-426b-8085-8fffb74a853e admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:06.371930 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0b45d657-acc2-426b-8085-8fffb74a853e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9fca31eaad0a453b9ee50c73fd5b99a5', 'request_id': 'req-0b45d657-acc2-426b-8085-8fffb74a853e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:06.377609 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0b45d657-acc2-426b-8085-8fffb74a853e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9fca31eaad0a453b9ee50c73fd5b99a5', 'request_id': 'req-0b45d657-acc2-426b-8085-8fffb74a853e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:06.383736 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0b45d657-acc2-426b-8085-8fffb74a853e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9fca31eaad0a453b9ee50c73fd5b99a5', 'request_id': 'req-0b45d657-acc2-426b-8085-8fffb74a853e'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:06.393582 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:06.398982 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0b45d657-acc2-426b-8085-8fffb74a853e admin admin] The token cache is being invalidate because project 9fca31eaad0a453b9ee50c73fd5b99a5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:27:06.399442 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0b45d657-acc2-426b-8085-8fffb74a853e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:06.400841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0b45d657-acc2-426b-8085-8fffb74a853e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:06.402038 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1426/2840] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:27:06 2022] DELETE /identity/v3/projects/9fca31eaad0a453b9ee50c73fd5b99a5 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:27:06.407380 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f908875-7896-4aa4-8154-f5d1b9fa2884 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:06.407936 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f908875-7896-4aa4-8154-f5d1b9fa2884 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:06.408210 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f908875-7896-4aa4-8154-f5d1b9fa2884 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:06.428164 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-0f908875-7896-4aa4-8154-f5d1b9fa2884 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:06.429618 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-0f908875-7896-4aa4-8154-f5d1b9fa2884 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:06.486550 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1415/2841] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:06.494476 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937bba97-5d96-4a87-bbe0-03b30b759bdd None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:06.494764 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937bba97-5d96-4a87-bbe0-03b30b759bdd None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:06.495074 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937bba97-5d96-4a87-bbe0-03b30b759bdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:06.514766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-937bba97-5d96-4a87-bbe0-03b30b759bdd None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:06.515977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-937bba97-5d96-4a87-bbe0-03b30b759bdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:06.524266 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1427/2842] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:06.528302 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f908875-7896-4aa4-8154-f5d1b9fa2884 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:06.537750 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f908875-7896-4aa4-8154-f5d1b9fa2884 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:06.542753 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da9d11a-0283-4c62-afd9-e0c444de4d58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:06.543367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da9d11a-0283-4c62-afd9-e0c444de4d58 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:06.543645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da9d11a-0283-4c62-afd9-e0c444de4d58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:06.543912 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da9d11a-0283-4c62-afd9-e0c444de4d58 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:06.544541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da9d11a-0283-4c62-afd9-e0c444de4d58 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:27:06.544843 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da9d11a-0283-4c62-afd9-e0c444de4d58 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:06.546408 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da9d11a-0283-4c62-afd9-e0c444de4d58 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:06.558024 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1416/2843] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:27:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:27:06.562464 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937bba97-5d96-4a87-bbe0-03b30b759bdd None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:06.572052 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937bba97-5d96-4a87-bbe0-03b30b759bdd None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:06.576662 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1954de7c-b18c-4371-b1ea-77bff35c6963 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:06.577246 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1954de7c-b18c-4371-b1ea-77bff35c6963 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:06.577522 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1954de7c-b18c-4371-b1ea-77bff35c6963 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:06.577793 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1954de7c-b18c-4371-b1ea-77bff35c6963 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:06.578454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1954de7c-b18c-4371-b1ea-77bff35c6963 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:06.580368 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1954de7c-b18c-4371-b1ea-77bff35c6963 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:06.606935 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1428/2844] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:27:06 2022] POST /identity/v3/projects => generated 375 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:27:06.610925 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da9d11a-0283-4c62-afd9-e0c444de4d58 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:06.620618 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da9d11a-0283-4c62-afd9-e0c444de4d58 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:06.669877 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:06.682024 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0186887e-b8bb-4c21-b318-865f45b98a1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:06.698638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0186887e-b8bb-4c21-b318-865f45b98a1c admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:06.698638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0186887e-b8bb-4c21-b318-865f45b98a1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:06.698638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0186887e-b8bb-4c21-b318-865f45b98a1c admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:06.698638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0186887e-b8bb-4c21-b318-865f45b98a1c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:06.698638 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0186887e-b8bb-4c21-b318-865f45b98a1c admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:06.713057 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1417/2845] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:27:06 2022] POST /identity/v3/users => generated 288 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:27:06.717182 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1954de7c-b18c-4371-b1ea-77bff35c6963 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:06.727975 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1954de7c-b18c-4371-b1ea-77bff35c6963 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:06.732502 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8991f52f-6304-41ae-ba40-935cfef32914 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:06.733097 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8991f52f-6304-41ae-ba40-935cfef32914 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:06.733374 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8991f52f-6304-41ae-ba40-935cfef32914 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:06.733648 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8991f52f-6304-41ae-ba40-935cfef32914 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:06.734282 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8991f52f-6304-41ae-ba40-935cfef32914 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:06.735896 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8991f52f-6304-41ae-ba40-935cfef32914 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:06.746294 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1429/2846] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:27:06 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:27:06.750130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0186887e-b8bb-4c21-b318-865f45b98a1c admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:06.759878 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0186887e-b8bb-4c21-b318-865f45b98a1c admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:06.764413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5de7ce0-34d3-4d03-9ed5-cd97e8e5d62f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:06.765016 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5de7ce0-34d3-4d03-9ed5-cd97e8e5d62f admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:06.765292 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5de7ce0-34d3-4d03-9ed5-cd97e8e5d62f admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:06.765559 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5de7ce0-34d3-4d03-9ed5-cd97e8e5d62f admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:06.766182 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5de7ce0-34d3-4d03-9ed5-cd97e8e5d62f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:06.767773 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5de7ce0-34d3-4d03-9ed5-cd97e8e5d62f admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:06.778170 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1418/2847] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:27:06 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:27:06.782177 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8991f52f-6304-41ae-ba40-935cfef32914 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:06.792425 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8991f52f-6304-41ae-ba40-935cfef32914 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:06.796965 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4546df1d-3249-47dd-b37f-b4c6de100082 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:06.797593 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4546df1d-3249-47dd-b37f-b4c6de100082 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:06.797872 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4546df1d-3249-47dd-b37f-b4c6de100082 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:06.798143 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4546df1d-3249-47dd-b37f-b4c6de100082 admin admin] PATH_INFO: `/v3/projects/76218bc5c68f4b9b8ad5a32db54b9484/users/1f418536df07498698842b6c24597f51/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:06.813127 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4546df1d-3249-47dd-b37f-b4c6de100082 admin admin] RBAC: Authorizing `identity:create_grant(project_id=76218bc5c68f4b9b8ad5a32db54b9484, user_id=1f418536df07498698842b6c24597f51, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:06.815781 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4546df1d-3249-47dd-b37f-b4c6de100082 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:06.828596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1430/2848] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:27:06 2022] PUT /identity/v3/projects/76218bc5c68f4b9b8ad5a32db54b9484/users/1f418536df07498698842b6c24597f51/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:27:06.841372 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43617c40-ed05-47a1-8b48-76bfcbb37959 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:06.841657 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43617c40-ed05-47a1-8b48-76bfcbb37959 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:06.841928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43617c40-ed05-47a1-8b48-76bfcbb37959 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:06.870436 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-43617c40-ed05-47a1-8b48-76bfcbb37959 None None] MFA Rules not processed for user `1f418536df07498698842b6c24597f51`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:06.871686 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-43617c40-ed05-47a1-8b48-76bfcbb37959 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:06.939072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1419/2849] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:06 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:06.949834 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4546df1d-3249-47dd-b37f-b4c6de100082 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:06.951533 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4546df1d-3249-47dd-b37f-b4c6de100082 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:07.011107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4546df1d-3249-47dd-b37f-b4c6de100082 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:07.014648 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78362871-e611-4d76-8bf4-4c465586404c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:07.015284 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78362871-e611-4d76-8bf4-4c465586404c service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:07.015565 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78362871-e611-4d76-8bf4-4c465586404c service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:07.015869 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78362871-e611-4d76-8bf4-4c465586404c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:07.024937 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78362871-e611-4d76-8bf4-4c465586404c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:07.026435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78362871-e611-4d76-8bf4-4c465586404c service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:07.035391 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1431/2850] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:27:06 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:27:14.060862 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43617c40-ed05-47a1-8b48-76bfcbb37959 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:14.062281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-43617c40-ed05-47a1-8b48-76bfcbb37959 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:14.131049 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43617c40-ed05-47a1-8b48-76bfcbb37959 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:14.136222 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8333d5-fc7c-4ac8-b532-bf0f7a5d825b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:14.136851 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8333d5-fc7c-4ac8-b532-bf0f7a5d825b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:14.137129 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8333d5-fc7c-4ac8-b532-bf0f7a5d825b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:14.137399 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8333d5-fc7c-4ac8-b532-bf0f7a5d825b admin admin] PATH_INFO: `/v3/users/0b18be1a47054a20a8de8d0343daefff` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:14.138618 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8333d5-fc7c-4ac8-b532-bf0f7a5d825b admin admin] RBAC: Authorizing `identity:delete_user(user_id=0b18be1a47054a20a8de8d0343daefff)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:14.140582 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8333d5-fc7c-4ac8-b532-bf0f7a5d825b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:14.189532 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ba8333d5-fc7c-4ac8-b532-bf0f7a5d825b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0b18be1a47054a20a8de8d0343daefff'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:14.190173 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ba8333d5-fc7c-4ac8-b532-bf0f7a5d825b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0b18be1a47054a20a8de8d0343daefff'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:14.198185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ba8333d5-fc7c-4ac8-b532-bf0f7a5d825b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0b18be1a47054a20a8de8d0343daefff'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:14.198872 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ba8333d5-fc7c-4ac8-b532-bf0f7a5d825b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0b18be1a47054a20a8de8d0343daefff'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:14.205724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-ba8333d5-fc7c-4ac8-b532-bf0f7a5d825b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0b18be1a47054a20a8de8d0343daefff'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:14.207163 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:14.226914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1420/2851] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:27:14 2022] DELETE /identity/v3/users/0b18be1a47054a20a8de8d0343daefff => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:27:14.234517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5cf5a9f-ca6c-4e0c-8aa2-711246ea1882 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:14.234517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5cf5a9f-ca6c-4e0c-8aa2-711246ea1882 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:14.234517 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5cf5a9f-ca6c-4e0c-8aa2-711246ea1882 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:14.251461 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e5cf5a9f-ca6c-4e0c-8aa2-711246ea1882 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:14.252618 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e5cf5a9f-ca6c-4e0c-8aa2-711246ea1882 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:14.314269 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1432/2852] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:14.324467 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8333d5-fc7c-4ac8-b532-bf0f7a5d825b admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:14.325878 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ba8333d5-fc7c-4ac8-b532-bf0f7a5d825b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:14.384616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8333d5-fc7c-4ac8-b532-bf0f7a5d825b admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:14.388334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88683d42-3fdc-469c-b228-5178de43e23b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:14.388947 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88683d42-3fdc-469c-b228-5178de43e23b service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:14.389269 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88683d42-3fdc-469c-b228-5178de43e23b service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:14.389494 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88683d42-3fdc-469c-b228-5178de43e23b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:14.399960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88683d42-3fdc-469c-b228-5178de43e23b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:14.401435 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88683d42-3fdc-469c-b228-5178de43e23b service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:14.416397 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1421/2853] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:27:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:27:14.688652 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5cf5a9f-ca6c-4e0c-8aa2-711246ea1882 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:14.690611 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e5cf5a9f-ca6c-4e0c-8aa2-711246ea1882 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:14.704924 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5cf5a9f-ca6c-4e0c-8aa2-711246ea1882 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:14.715844 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fae9961-94f6-4a70-a78f-12029a0552e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:14.716549 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fae9961-94f6-4a70-a78f-12029a0552e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:14.716998 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fae9961-94f6-4a70-a78f-12029a0552e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:14.717177 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fae9961-94f6-4a70-a78f-12029a0552e0 admin admin] PATH_INFO: `/v3/projects/3d8d0a19f1934a49aa1e0fb7ccfaa5aa` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:14.718422 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fae9961-94f6-4a70-a78f-12029a0552e0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3d8d0a19f1934a49aa1e0fb7ccfaa5aa)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:14.720453 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fae9961-94f6-4a70-a78f-12029a0552e0 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:14.763595 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:14.781150 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0fae9961-94f6-4a70-a78f-12029a0552e0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3d8d0a19f1934a49aa1e0fb7ccfaa5aa', 'request_id': 'req-0fae9961-94f6-4a70-a78f-12029a0552e0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:14.787167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0fae9961-94f6-4a70-a78f-12029a0552e0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3d8d0a19f1934a49aa1e0fb7ccfaa5aa', 'request_id': 'req-0fae9961-94f6-4a70-a78f-12029a0552e0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:14.796378 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0fae9961-94f6-4a70-a78f-12029a0552e0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3d8d0a19f1934a49aa1e0fb7ccfaa5aa', 'request_id': 'req-0fae9961-94f6-4a70-a78f-12029a0552e0'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:14.811126 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0fae9961-94f6-4a70-a78f-12029a0552e0 admin admin] The token cache is being invalidate because project 3d8d0a19f1934a49aa1e0fb7ccfaa5aa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:27:14.811412 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0fae9961-94f6-4a70-a78f-12029a0552e0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:14.812107 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0fae9961-94f6-4a70-a78f-12029a0552e0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:14.813720 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1433/2854] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:27:14 2022] DELETE /identity/v3/projects/3d8d0a19f1934a49aa1e0fb7ccfaa5aa => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:27:14.818590 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdba5537-1a85-418f-b8c5-70f85bfb50ea None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:14.818964 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdba5537-1a85-418f-b8c5-70f85bfb50ea None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:14.819323 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdba5537-1a85-418f-b8c5-70f85bfb50ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:14.842903 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-cdba5537-1a85-418f-b8c5-70f85bfb50ea None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:14.846083 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-cdba5537-1a85-418f-b8c5-70f85bfb50ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:14.908946 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1422/2855] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:27:14.917304 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48b3e2a-f16d-40b5-b6d4-cb4abd773712 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:14.917971 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48b3e2a-f16d-40b5-b6d4-cb4abd773712 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:14.917971 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48b3e2a-f16d-40b5-b6d4-cb4abd773712 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:14.939946 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-c48b3e2a-f16d-40b5-b6d4-cb4abd773712 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:14.941397 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c48b3e2a-f16d-40b5-b6d4-cb4abd773712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:14.949557 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1434/2856] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:14.953773 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdba5537-1a85-418f-b8c5-70f85bfb50ea None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:14.963624 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdba5537-1a85-418f-b8c5-70f85bfb50ea None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:14.968266 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93c57a2-0bda-4bc8-a4e1-63e636350923 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:14.968868 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93c57a2-0bda-4bc8-a4e1-63e636350923 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:14.969149 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93c57a2-0bda-4bc8-a4e1-63e636350923 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:14.969425 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93c57a2-0bda-4bc8-a4e1-63e636350923 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:14.970060 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93c57a2-0bda-4bc8-a4e1-63e636350923 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:27:14.970365 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93c57a2-0bda-4bc8-a4e1-63e636350923 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:14.971988 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93c57a2-0bda-4bc8-a4e1-63e636350923 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:14.984856 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1423/2857] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:27:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:27:14.989049 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48b3e2a-f16d-40b5-b6d4-cb4abd773712 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:14.999218 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48b3e2a-f16d-40b5-b6d4-cb4abd773712 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:15.003911 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51d001f-f32e-481c-acbd-051ffe97faee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:15.004515 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51d001f-f32e-481c-acbd-051ffe97faee admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:15.004816 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51d001f-f32e-481c-acbd-051ffe97faee admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:15.005090 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51d001f-f32e-481c-acbd-051ffe97faee admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:15.005774 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51d001f-f32e-481c-acbd-051ffe97faee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:15.007723 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51d001f-f32e-481c-acbd-051ffe97faee admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:15.037497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1435/2858] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:27:14 2022] POST /identity/v3/projects => generated 369 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:27:15.042368 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93c57a2-0bda-4bc8-a4e1-63e636350923 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:15.052839 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93c57a2-0bda-4bc8-a4e1-63e636350923 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:15.115011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69dfdae-8004-4b55-b5f5-56811ef5ebb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:15.115708 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69dfdae-8004-4b55-b5f5-56811ef5ebb2 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:15.115921 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69dfdae-8004-4b55-b5f5-56811ef5ebb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:15.116148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69dfdae-8004-4b55-b5f5-56811ef5ebb2 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:15.117318 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69dfdae-8004-4b55-b5f5-56811ef5ebb2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:15.119236 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69dfdae-8004-4b55-b5f5-56811ef5ebb2 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:15.137778 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:15.144076 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1424/2859] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:27:15 2022] POST /identity/v3/users => generated 285 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:27:15.148644 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51d001f-f32e-481c-acbd-051ffe97faee admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:15.159414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51d001f-f32e-481c-acbd-051ffe97faee admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:15.164289 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0aeff48-d0ff-4b16-972d-478b7324452f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:15.164889 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0aeff48-d0ff-4b16-972d-478b7324452f admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:15.165279 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0aeff48-d0ff-4b16-972d-478b7324452f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:15.165552 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0aeff48-d0ff-4b16-972d-478b7324452f admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:15.166211 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0aeff48-d0ff-4b16-972d-478b7324452f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:15.168497 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0aeff48-d0ff-4b16-972d-478b7324452f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:15.186340 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1436/2860] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:27:15 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:27:15.188077 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69dfdae-8004-4b55-b5f5-56811ef5ebb2 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:15.200735 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69dfdae-8004-4b55-b5f5-56811ef5ebb2 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:15.206587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ae451f-62fb-480b-b249-2f6c0fdd44ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:15.207083 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ae451f-62fb-480b-b249-2f6c0fdd44ea admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:15.207363 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ae451f-62fb-480b-b249-2f6c0fdd44ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:15.207636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ae451f-62fb-480b-b249-2f6c0fdd44ea admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:15.208265 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ae451f-62fb-480b-b249-2f6c0fdd44ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:15.209881 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ae451f-62fb-480b-b249-2f6c0fdd44ea admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:15.221121 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1425/2861] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:27:15 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:27:15.225089 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0aeff48-d0ff-4b16-972d-478b7324452f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:15.235569 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0aeff48-d0ff-4b16-972d-478b7324452f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:15.240410 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d408c8f9-02be-4991-a163-5af45294640a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:15.241060 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d408c8f9-02be-4991-a163-5af45294640a admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:15.241865 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d408c8f9-02be-4991-a163-5af45294640a admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:15.242135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d408c8f9-02be-4991-a163-5af45294640a admin admin] PATH_INFO: `/v3/projects/a606ddb7c1b346889111a569520789a9/users/8e2e08649e4445b88d156f3411364186/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:15.256787 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d408c8f9-02be-4991-a163-5af45294640a admin admin] RBAC: Authorizing `identity:create_grant(project_id=a606ddb7c1b346889111a569520789a9, user_id=8e2e08649e4445b88d156f3411364186, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:15.260146 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d408c8f9-02be-4991-a163-5af45294640a admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:15.275019 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1437/2862] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:27:15 2022] PUT /identity/v3/projects/a606ddb7c1b346889111a569520789a9/users/8e2e08649e4445b88d156f3411364186/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:27:15.287723 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feef3b61-6fdf-43fa-9fff-d69afac045cf None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:15.287944 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feef3b61-6fdf-43fa-9fff-d69afac045cf None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:15.288263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feef3b61-6fdf-43fa-9fff-d69afac045cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:15.316346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-feef3b61-6fdf-43fa-9fff-d69afac045cf None None] MFA Rules not processed for user `8e2e08649e4445b88d156f3411364186`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:15.317554 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-feef3b61-6fdf-43fa-9fff-d69afac045cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:15.382887 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1426/2863] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:15 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:15.394608 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d408c8f9-02be-4991-a163-5af45294640a admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:15.396488 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-d408c8f9-02be-4991-a163-5af45294640a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:15.483814 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d408c8f9-02be-4991-a163-5af45294640a admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:15.487621 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849d840e-d899-4b6f-a1ec-124f1182952d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eef98bcf2699461d9a7f3c3bffe82f34', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:15.488331 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849d840e-d899-4b6f-a1ec-124f1182952d service glance] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:15.488700 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849d840e-d899-4b6f-a1ec-124f1182952d service glance] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:15.488974 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849d840e-d899-4b6f-a1ec-124f1182952d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:15.497512 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849d840e-d899-4b6f-a1ec-124f1182952d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:15.498962 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849d840e-d899-4b6f-a1ec-124f1182952d service glance] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:15.507848 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1438/2864] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:27:15 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:27:19.216965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feef3b61-6fdf-43fa-9fff-d69afac045cf None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:19.218352 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-feef3b61-6fdf-43fa-9fff-d69afac045cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:19.282845 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feef3b61-6fdf-43fa-9fff-d69afac045cf None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:19.287333 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cfdd42-2fcb-4835-9864-ccdd6028960a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:19.288640 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cfdd42-2fcb-4835-9864-ccdd6028960a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:19.288947 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cfdd42-2fcb-4835-9864-ccdd6028960a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:19.289220 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cfdd42-2fcb-4835-9864-ccdd6028960a admin admin] PATH_INFO: `/v3/users/1f418536df07498698842b6c24597f51` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:19.290327 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cfdd42-2fcb-4835-9864-ccdd6028960a admin admin] RBAC: Authorizing `identity:delete_user(user_id=1f418536df07498698842b6c24597f51)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:19.292290 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cfdd42-2fcb-4835-9864-ccdd6028960a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:19.338786 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-01cfdd42-2fcb-4835-9864-ccdd6028960a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1f418536df07498698842b6c24597f51'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:19.339376 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-01cfdd42-2fcb-4835-9864-ccdd6028960a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1f418536df07498698842b6c24597f51'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:19.346228 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-01cfdd42-2fcb-4835-9864-ccdd6028960a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1f418536df07498698842b6c24597f51'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:19.346834 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-01cfdd42-2fcb-4835-9864-ccdd6028960a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1f418536df07498698842b6c24597f51'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:19.353646 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-01cfdd42-2fcb-4835-9864-ccdd6028960a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1f418536df07498698842b6c24597f51'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:19.366010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:19.371794 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1427/2865] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:27:19 2022] DELETE /identity/v3/users/1f418536df07498698842b6c24597f51 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:27:19.376506 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbedde85-e995-4e2f-a6b4-6086faf42c19 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:19.377147 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbedde85-e995-4e2f-a6b4-6086faf42c19 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:19.377147 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbedde85-e995-4e2f-a6b4-6086faf42c19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:19.397328 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-dbedde85-e995-4e2f-a6b4-6086faf42c19 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:19.399050 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-dbedde85-e995-4e2f-a6b4-6086faf42c19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:19.464887 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1439/2866] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:19.474612 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cfdd42-2fcb-4835-9864-ccdd6028960a admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:19.476114 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-01cfdd42-2fcb-4835-9864-ccdd6028960a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:19.534168 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cfdd42-2fcb-4835-9864-ccdd6028960a admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:19.537744 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df171720-c81a-44ad-88e4-d2cfd4cbac85 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:19.538362 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df171720-c81a-44ad-88e4-d2cfd4cbac85 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:19.538640 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df171720-c81a-44ad-88e4-d2cfd4cbac85 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:19.538947 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df171720-c81a-44ad-88e4-d2cfd4cbac85 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:19.548010 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df171720-c81a-44ad-88e4-d2cfd4cbac85 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:19.549437 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df171720-c81a-44ad-88e4-d2cfd4cbac85 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:19.561078 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1428/2867] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:27:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:27:19.836421 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbedde85-e995-4e2f-a6b4-6086faf42c19 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:19.837918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-dbedde85-e995-4e2f-a6b4-6086faf42c19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:19.844994 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:DELETE=5,keystone:SELECT=92,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:19.851821 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbedde85-e995-4e2f-a6b4-6086faf42c19 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:19.857943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d385e6-33d2-40ab-b1e8-04c222d58a22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:19.858672 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d385e6-33d2-40ab-b1e8-04c222d58a22 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:19.859226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d385e6-33d2-40ab-b1e8-04c222d58a22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:19.860253 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d385e6-33d2-40ab-b1e8-04c222d58a22 admin admin] PATH_INFO: `/v3/projects/76218bc5c68f4b9b8ad5a32db54b9484` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:19.861864 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d385e6-33d2-40ab-b1e8-04c222d58a22 admin admin] RBAC: Authorizing `identity:delete_project(project_id=76218bc5c68f4b9b8ad5a32db54b9484)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:19.864324 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d385e6-33d2-40ab-b1e8-04c222d58a22 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:19.915364 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-02d385e6-33d2-40ab-b1e8-04c222d58a22 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '76218bc5c68f4b9b8ad5a32db54b9484', 'request_id': 'req-02d385e6-33d2-40ab-b1e8-04c222d58a22'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:19.921103 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-02d385e6-33d2-40ab-b1e8-04c222d58a22 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '76218bc5c68f4b9b8ad5a32db54b9484', 'request_id': 'req-02d385e6-33d2-40ab-b1e8-04c222d58a22'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:19.927375 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-02d385e6-33d2-40ab-b1e8-04c222d58a22 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '76218bc5c68f4b9b8ad5a32db54b9484', 'request_id': 'req-02d385e6-33d2-40ab-b1e8-04c222d58a22'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:19.940317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-02d385e6-33d2-40ab-b1e8-04c222d58a22 admin admin] The token cache is being invalidate because project 76218bc5c68f4b9b8ad5a32db54b9484 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:27:19.940609 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-02d385e6-33d2-40ab-b1e8-04c222d58a22 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:19.941243 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-02d385e6-33d2-40ab-b1e8-04c222d58a22 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:19.942722 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1440/2868] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:27:19 2022] DELETE /identity/v3/projects/76218bc5c68f4b9b8ad5a32db54b9484 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:27:19.948353 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193fc5fb-99b2-4e29-8e82-bcddd0a0d539 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:19.948644 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193fc5fb-99b2-4e29-8e82-bcddd0a0d539 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:19.948942 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193fc5fb-99b2-4e29-8e82-bcddd0a0d539 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:19.969255 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-193fc5fb-99b2-4e29-8e82-bcddd0a0d539 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:19.970474 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-193fc5fb-99b2-4e29-8e82-bcddd0a0d539 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:20.029027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1429/2869] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:20.093547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41b5645-1d5b-48c8-b225-8483bc4d8133 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:20.093836 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41b5645-1d5b-48c8-b225-8483bc4d8133 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:20.094134 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41b5645-1d5b-48c8-b225-8483bc4d8133 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:20.119222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-f41b5645-1d5b-48c8-b225-8483bc4d8133 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:20.120536 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f41b5645-1d5b-48c8-b225-8483bc4d8133 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:20.129147 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1441/2870] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:20.133653 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193fc5fb-99b2-4e29-8e82-bcddd0a0d539 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:20.143777 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193fc5fb-99b2-4e29-8e82-bcddd0a0d539 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:20.148395 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0ca493-74a0-4869-9801-46bff7a39016 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:20.149023 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0ca493-74a0-4869-9801-46bff7a39016 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:20.149300 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0ca493-74a0-4869-9801-46bff7a39016 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:20.149571 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0ca493-74a0-4869-9801-46bff7a39016 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:20.150235 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0ca493-74a0-4869-9801-46bff7a39016 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:27:20.150538 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0ca493-74a0-4869-9801-46bff7a39016 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:20.152155 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0ca493-74a0-4869-9801-46bff7a39016 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:20.164403 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1430/2871] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:27:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 01:27:20.169260 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41b5645-1d5b-48c8-b225-8483bc4d8133 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:20.179564 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41b5645-1d5b-48c8-b225-8483bc4d8133 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:20.187255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36c060a-c018-44e8-8d02-6faaef1234ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:20.187255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36c060a-c018-44e8-8d02-6faaef1234ce admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:20.187255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36c060a-c018-44e8-8d02-6faaef1234ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:20.187255 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36c060a-c018-44e8-8d02-6faaef1234ce admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:20.187614 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36c060a-c018-44e8-8d02-6faaef1234ce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:20.188526 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36c060a-c018-44e8-8d02-6faaef1234ce admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:20.218890 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1442/2872] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:27:20 2022] POST /identity/v3/projects => generated 383 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:27:20.222713 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0ca493-74a0-4869-9801-46bff7a39016 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:20.233194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0ca493-74a0-4869-9801-46bff7a39016 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:20.289375 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1094fad6-4690-4f77-9e62-f448f37e15ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:20.289965 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1094fad6-4690-4f77-9e62-f448f37e15ab admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:20.290230 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1094fad6-4690-4f77-9e62-f448f37e15ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:20.290499 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1094fad6-4690-4f77-9e62-f448f37e15ab admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:20.291137 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1094fad6-4690-4f77-9e62-f448f37e15ab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:20.293029 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1094fad6-4690-4f77-9e62-f448f37e15ab admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:20.313645 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1431/2873] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:27:20 2022] POST /identity/v3/users => generated 292 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:27:20.317806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36c060a-c018-44e8-8d02-6faaef1234ce admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:20.328431 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36c060a-c018-44e8-8d02-6faaef1234ce admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:20.333091 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bc51e6-31ac-40d9-b77d-0afa03957a07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:20.333688 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bc51e6-31ac-40d9-b77d-0afa03957a07 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:20.333969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bc51e6-31ac-40d9-b77d-0afa03957a07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:20.334244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bc51e6-31ac-40d9-b77d-0afa03957a07 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:20.334914 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bc51e6-31ac-40d9-b77d-0afa03957a07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:20.336492 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bc51e6-31ac-40d9-b77d-0afa03957a07 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:20.347973 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1443/2874] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:27:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:27:20.351541 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1094fad6-4690-4f77-9e62-f448f37e15ab admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:20.361738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1094fad6-4690-4f77-9e62-f448f37e15ab admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:20.366920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13715d6-e74d-4a0d-b3f5-a5ed118b338a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:20.367515 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13715d6-e74d-4a0d-b3f5-a5ed118b338a admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:20.367787 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13715d6-e74d-4a0d-b3f5-a5ed118b338a admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:20.368050 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13715d6-e74d-4a0d-b3f5-a5ed118b338a admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:20.368671 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13715d6-e74d-4a0d-b3f5-a5ed118b338a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:20.370245 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13715d6-e74d-4a0d-b3f5-a5ed118b338a admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:20.382037 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1432/2875] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:27:20 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:27:20.386186 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bc51e6-31ac-40d9-b77d-0afa03957a07 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:20.396262 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bc51e6-31ac-40d9-b77d-0afa03957a07 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:20.401472 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cbff21-627c-4445-86a4-9c0b6a926394 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:20.402258 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cbff21-627c-4445-86a4-9c0b6a926394 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:20.402538 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cbff21-627c-4445-86a4-9c0b6a926394 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:20.402806 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cbff21-627c-4445-86a4-9c0b6a926394 admin admin] PATH_INFO: `/v3/projects/7bede684fde243038bcb6b0d5c31ac8f/users/d54805410115495597ef0720af2acf22/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:20.426826 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68cbff21-627c-4445-86a4-9c0b6a926394 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7bede684fde243038bcb6b0d5c31ac8f, user_id=d54805410115495597ef0720af2acf22, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:20.429791 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68cbff21-627c-4445-86a4-9c0b6a926394 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:20.443168 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1444/2876] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:27:20 2022] PUT /identity/v3/projects/7bede684fde243038bcb6b0d5c31ac8f/users/d54805410115495597ef0720af2acf22/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:27:20.448201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cd0671-1981-4fd5-9d9f-59ab9034c475 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:20.448491 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cd0671-1981-4fd5-9d9f-59ab9034c475 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:20.448768 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cd0671-1981-4fd5-9d9f-59ab9034c475 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:20.462361 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:20.468816 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-29cd0671-1981-4fd5-9d9f-59ab9034c475 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:20.470184 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-29cd0671-1981-4fd5-9d9f-59ab9034c475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:20.533201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1433/2877] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:27:20.542874 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cbff21-627c-4445-86a4-9c0b6a926394 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:20.544609 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-68cbff21-627c-4445-86a4-9c0b6a926394 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:20.603523 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cbff21-627c-4445-86a4-9c0b6a926394 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:20.607234 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f694a2-4cb3-4944-a9d3-89b6041f7eea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:20.607841 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f694a2-4cb3-4944-a9d3-89b6041f7eea service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:20.608118 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f694a2-4cb3-4944-a9d3-89b6041f7eea service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:20.608392 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f694a2-4cb3-4944-a9d3-89b6041f7eea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:20.617860 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f694a2-4cb3-4944-a9d3-89b6041f7eea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:20.619357 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f694a2-4cb3-4944-a9d3-89b6041f7eea service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:20.628696 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1445/2878] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:27:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:27:24.703463 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1383d425-d723-403a-9b8b-1b9355416e6b None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:24.704114 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1383d425-d723-403a-9b8b-1b9355416e6b None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:24.704391 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1383d425-d723-403a-9b8b-1b9355416e6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:24.735766 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-1383d425-d723-403a-9b8b-1b9355416e6b None None] MFA Rules not processed for user `d54805410115495597ef0720af2acf22`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:24.737627 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1383d425-d723-403a-9b8b-1b9355416e6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:24.820262 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1434/2879] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:24 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:24.824820 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f694a2-4cb3-4944-a9d3-89b6041f7eea service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:24.835827 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f694a2-4cb3-4944-a9d3-89b6041f7eea service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:24.841161 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9129db3e-4d41-4185-8b75-ed941c1ab888 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:24.841786 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9129db3e-4d41-4185-8b75-ed941c1ab888 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:24.842085 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9129db3e-4d41-4185-8b75-ed941c1ab888 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:24.842358 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9129db3e-4d41-4185-8b75-ed941c1ab888 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:24.843035 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9129db3e-4d41-4185-8b75-ed941c1ab888 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:24.844968 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9129db3e-4d41-4185-8b75-ed941c1ab888 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:24.873583 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1446/2880] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:27:24 2022] POST /identity/v3/projects => generated 385 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:27:24.877413 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1383d425-d723-403a-9b8b-1b9355416e6b None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:24.889614 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1383d425-d723-403a-9b8b-1b9355416e6b None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:24.952786 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f1bce4-8e44-4e88-a5c8-8a803c9a2ad8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:24.953749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f1bce4-8e44-4e88-a5c8-8a803c9a2ad8 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:24.954032 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f1bce4-8e44-4e88-a5c8-8a803c9a2ad8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:24.954306 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f1bce4-8e44-4e88-a5c8-8a803c9a2ad8 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:24.955791 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f1bce4-8e44-4e88-a5c8-8a803c9a2ad8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:24.957738 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f1bce4-8e44-4e88-a5c8-8a803c9a2ad8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:24.979920 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1435/2881] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:27:24 2022] POST /identity/v3/users => generated 299 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:27:24.984026 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9129db3e-4d41-4185-8b75-ed941c1ab888 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:24.989837 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:24.996248 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9129db3e-4d41-4185-8b75-ed941c1ab888 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:25.007943 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd18476-6ca7-4d5a-9d9e-dc4d818030d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:25.008681 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd18476-6ca7-4d5a-9d9e-dc4d818030d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:25.008963 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd18476-6ca7-4d5a-9d9e-dc4d818030d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:25.009266 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd18476-6ca7-4d5a-9d9e-dc4d818030d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:25.009913 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd18476-6ca7-4d5a-9d9e-dc4d818030d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:25.011578 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd18476-6ca7-4d5a-9d9e-dc4d818030d3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:25.026997 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1447/2882] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:27:24 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:27:25.028947 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f1bce4-8e44-4e88-a5c8-8a803c9a2ad8 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:25.040098 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f1bce4-8e44-4e88-a5c8-8a803c9a2ad8 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:25.044849 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ef17f6-d91e-4be6-a81c-c4f7b5fb2ec8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:25.045520 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ef17f6-d91e-4be6-a81c-c4f7b5fb2ec8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:25.045801 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ef17f6-d91e-4be6-a81c-c4f7b5fb2ec8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:25.046076 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ef17f6-d91e-4be6-a81c-c4f7b5fb2ec8 admin admin] PATH_INFO: `/v3/projects/bcbe9f2e8f6a4faeb418d1bbc59a47dc/users/7b1e39df9fd8471f9f36e52e2a1f7f0c/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:25.080875 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ef17f6-d91e-4be6-a81c-c4f7b5fb2ec8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bcbe9f2e8f6a4faeb418d1bbc59a47dc, user_id=7b1e39df9fd8471f9f36e52e2a1f7f0c, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:25.083621 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ef17f6-d91e-4be6-a81c-c4f7b5fb2ec8 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:25.097830 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1436/2883] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:27:25 2022] PUT /identity/v3/projects/bcbe9f2e8f6a4faeb418d1bbc59a47dc/users/7b1e39df9fd8471f9f36e52e2a1f7f0c/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:27:25.101812 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd18476-6ca7-4d5a-9d9e-dc4d818030d3 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:25.112655 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd18476-6ca7-4d5a-9d9e-dc4d818030d3 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:25.186213 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1beef9-350a-45d4-86f1-ed831ec6e845 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:25.186833 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1beef9-350a-45d4-86f1-ed831ec6e845 admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:25.187163 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1beef9-350a-45d4-86f1-ed831ec6e845 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:25.187988 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1beef9-350a-45d4-86f1-ed831ec6e845 admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:25.188371 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1beef9-350a-45d4-86f1-ed831ec6e845 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:25.189825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1beef9-350a-45d4-86f1-ed831ec6e845 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:25.202899 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1448/2884] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:27:25 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:27:25.205749 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ef17f6-d91e-4be6-a81c-c4f7b5fb2ec8 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:25.211928 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:25.218236 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ef17f6-d91e-4be6-a81c-c4f7b5fb2ec8 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:25.224716 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f489f5-9d11-41a6-bdf5-2b49003f0430 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:25.225496 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f489f5-9d11-41a6-bdf5-2b49003f0430 admin admin] REQUEST_METHOD: `PUT` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:25.225850 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f489f5-9d11-41a6-bdf5-2b49003f0430 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:25.226193 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f489f5-9d11-41a6-bdf5-2b49003f0430 admin admin] PATH_INFO: `/v3/projects/bcbe9f2e8f6a4faeb418d1bbc59a47dc/users/7b1e39df9fd8471f9f36e52e2a1f7f0c/roles/66a36c858aab40688080eed6ceace3a0` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:25.228312 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f489f5-9d11-41a6-bdf5-2b49003f0430 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bcbe9f2e8f6a4faeb418d1bbc59a47dc, user_id=7b1e39df9fd8471f9f36e52e2a1f7f0c, role_id=66a36c858aab40688080eed6ceace3a0)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:25.231149 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f489f5-9d11-41a6-bdf5-2b49003f0430 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:25.242834 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1437/2885] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:27:25 2022] PUT /identity/v3/projects/bcbe9f2e8f6a4faeb418d1bbc59a47dc/users/7b1e39df9fd8471f9f36e52e2a1f7f0c/roles/66a36c858aab40688080eed6ceace3a0 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:27:29.681212 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dd5654-58a0-4df1-9727-7d5f512dd772 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:29.681749 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dd5654-58a0-4df1-9727-7d5f512dd772 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:29.682000 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dd5654-58a0-4df1-9727-7d5f512dd772 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:29.714993 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-e2dd5654-58a0-4df1-9727-7d5f512dd772 None None] MFA Rules not processed for user `7b1e39df9fd8471f9f36e52e2a1f7f0c`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:29.716451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e2dd5654-58a0-4df1-9727-7d5f512dd772 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:29.826189 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1449/2886] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:29 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:29.840960 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f489f5-9d11-41a6-bdf5-2b49003f0430 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:29.843958 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-15f489f5-9d11-41a6-bdf5-2b49003f0430 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:29.963683 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f489f5-9d11-41a6-bdf5-2b49003f0430 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:29.969240 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf517bf-770b-49a9-bc5f-d24c9070c88f service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eef98bcf2699461d9a7f3c3bffe82f34', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:29.970740 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf517bf-770b-49a9-bc5f-d24c9070c88f service glance] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:29.971201 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf517bf-770b-49a9-bc5f-d24c9070c88f service glance] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:29.971927 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf517bf-770b-49a9-bc5f-d24c9070c88f service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:30.058730 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf517bf-770b-49a9-bc5f-d24c9070c88f service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:30.061270 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf517bf-770b-49a9-bc5f-d24c9070c88f service glance] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:30.079096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1438/2887] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:27:29 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:27:30.226682 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af881ad-1596-43d6-8633-0ce0af4ed4f5 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:30.227277 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af881ad-1596-43d6-8633-0ce0af4ed4f5 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:30.227764 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af881ad-1596-43d6-8633-0ce0af4ed4f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:30.265891 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-4af881ad-1596-43d6-8633-0ce0af4ed4f5 None None] MFA Rules not processed for user `e28a0ba45389417ba8ae92376dba87ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:30.267675 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-4af881ad-1596-43d6-8633-0ce0af4ed4f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:30.341614 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1450/2888] 10.222.0.15 () {58 vars in 1009 bytes} [Tue Sep 13 01:27:30 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:30.394649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf517bf-770b-49a9-bc5f-d24c9070c88f service glance] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:30.396127 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fbf517bf-770b-49a9-bc5f-d24c9070c88f service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:30.481019 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf517bf-770b-49a9-bc5f-d24c9070c88f service glance] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:30.485997 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9885101-32f5-4d46-b0d9-c1155e6f3175 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d7d8ed4ede4ddca192fce0a497689d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:30.486602 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9885101-32f5-4d46-b0d9-c1155e6f3175 service swift] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:30.486914 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9885101-32f5-4d46-b0d9-c1155e6f3175 service swift] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:30.487198 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9885101-32f5-4d46-b0d9-c1155e6f3175 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:30.496926 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9885101-32f5-4d46-b0d9-c1155e6f3175 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:30.498564 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9885101-32f5-4d46-b0d9-c1155e6f3175 service swift] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:30.512433 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1439/2889] 10.222.0.15 () {60 vars in 1384 bytes} [Tue Sep 13 01:27:30 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 01:27:39.284402 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee71d32f-710b-4cd8-a796-6b11292bd516 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:39.284916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee71d32f-710b-4cd8-a796-6b11292bd516 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:39.285029 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee71d32f-710b-4cd8-a796-6b11292bd516 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:39.292704 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=100 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:39.305624 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-ee71d32f-710b-4cd8-a796-6b11292bd516 None None] MFA Rules not processed for user `d54805410115495597ef0720af2acf22`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:39.306945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-ee71d32f-710b-4cd8-a796-6b11292bd516 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:39.313746 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1451/2890] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:39 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:40.509281 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=84,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:41.235151 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e30c1c2-8196-4707-a099-8d4dd9f7eae5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:41.235438 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e30c1c2-8196-4707-a099-8d4dd9f7eae5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:41.235780 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e30c1c2-8196-4707-a099-8d4dd9f7eae5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:41.256946 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-5e30c1c2-8196-4707-a099-8d4dd9f7eae5 None None] MFA Rules not processed for user `e28a0ba45389417ba8ae92376dba87ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:41.258434 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5e30c1c2-8196-4707-a099-8d4dd9f7eae5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:41.265980 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1440/2891] 10.222.0.15 () {58 vars in 1009 bytes} [Tue Sep 13 01:27:41 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:41.278965 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee71d32f-710b-4cd8-a796-6b11292bd516 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:41.290758 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee71d32f-710b-4cd8-a796-6b11292bd516 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:41.295720 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0465be-162b-474c-a00d-d632a0d14ef4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d7d8ed4ede4ddca192fce0a497689d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:41.296317 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0465be-162b-474c-a00d-d632a0d14ef4 service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:41.296596 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0465be-162b-474c-a00d-d632a0d14ef4 service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:41.296869 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0465be-162b-474c-a00d-d632a0d14ef4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:41.305121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c0465be-162b-474c-a00d-d632a0d14ef4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:41.306712 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c0465be-162b-474c-a00d-d632a0d14ef4 service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:41.314909 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1452/2892] 10.222.0.15 () {60 vars in 1384 bytes} [Tue Sep 13 01:27:41 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 01:27:41.813179 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e30c1c2-8196-4707-a099-8d4dd9f7eae5 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:41.815610 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-5e30c1c2-8196-4707-a099-8d4dd9f7eae5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:41.967159 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e30c1c2-8196-4707-a099-8d4dd9f7eae5 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:41.973521 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9349fc71-fab5-49bc-9d98-029ef8e61e52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:41.974801 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9349fc71-fab5-49bc-9d98-029ef8e61e52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:41.975130 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9349fc71-fab5-49bc-9d98-029ef8e61e52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:41.975471 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9349fc71-fab5-49bc-9d98-029ef8e61e52 admin admin] PATH_INFO: `/v3/users/8e2e08649e4445b88d156f3411364186` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:41.976697 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9349fc71-fab5-49bc-9d98-029ef8e61e52 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8e2e08649e4445b88d156f3411364186)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:41.978915 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9349fc71-fab5-49bc-9d98-029ef8e61e52 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:42.036284 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9349fc71-fab5-49bc-9d98-029ef8e61e52 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8e2e08649e4445b88d156f3411364186'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:42.037823 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9349fc71-fab5-49bc-9d98-029ef8e61e52 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8e2e08649e4445b88d156f3411364186'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:42.046319 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9349fc71-fab5-49bc-9d98-029ef8e61e52 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8e2e08649e4445b88d156f3411364186'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:42.047390 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9349fc71-fab5-49bc-9d98-029ef8e61e52 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8e2e08649e4445b88d156f3411364186'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:42.060378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-9349fc71-fab5-49bc-9d98-029ef8e61e52 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8e2e08649e4445b88d156f3411364186'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:42.085597 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1441/2893] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:27:41 2022] DELETE /identity/v3/users/8e2e08649e4445b88d156f3411364186 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:27:42.090182 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a4cd62-39c1-4626-9496-ec91f28a66c8 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:42.090534 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a4cd62-39c1-4626-9496-ec91f28a66c8 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:42.091192 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a4cd62-39c1-4626-9496-ec91f28a66c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:42.115349 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-91a4cd62-39c1-4626-9496-ec91f28a66c8 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:42.120433 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-91a4cd62-39c1-4626-9496-ec91f28a66c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:42.215167 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1453/2894] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:27:42.225211 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9349fc71-fab5-49bc-9d98-029ef8e61e52 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:42.227056 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-9349fc71-fab5-49bc-9d98-029ef8e61e52 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:42.327642 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9349fc71-fab5-49bc-9d98-029ef8e61e52 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:42.331333 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c410ea9e-6677-4275-b0d8-37a6facafd10 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:42.331939 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c410ea9e-6677-4275-b0d8-37a6facafd10 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:42.332228 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c410ea9e-6677-4275-b0d8-37a6facafd10 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:42.332481 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c410ea9e-6677-4275-b0d8-37a6facafd10 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:42.342669 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c410ea9e-6677-4275-b0d8-37a6facafd10 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:42.344134 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c410ea9e-6677-4275-b0d8-37a6facafd10 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:42.355440 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1442/2895] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:27:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 01:27:42.660209 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a4cd62-39c1-4626-9496-ec91f28a66c8 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:42.662437 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-91a4cd62-39c1-4626-9496-ec91f28a66c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:42.674612 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a4cd62-39c1-4626-9496-ec91f28a66c8 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:42.680351 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bed543-c5dd-4eb1-bf65-a0c6a82cb956 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:42.681137 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bed543-c5dd-4eb1-bf65-a0c6a82cb956 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:42.681482 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bed543-c5dd-4eb1-bf65-a0c6a82cb956 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:42.681825 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bed543-c5dd-4eb1-bf65-a0c6a82cb956 admin admin] PATH_INFO: `/v3/projects/a606ddb7c1b346889111a569520789a9` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:42.683136 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bed543-c5dd-4eb1-bf65-a0c6a82cb956 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a606ddb7c1b346889111a569520789a9)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:42.685244 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bed543-c5dd-4eb1-bf65-a0c6a82cb956 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:42.751064 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a2bed543-c5dd-4eb1-bf65-a0c6a82cb956 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a606ddb7c1b346889111a569520789a9', 'request_id': 'req-a2bed543-c5dd-4eb1-bf65-a0c6a82cb956'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:42.758311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a2bed543-c5dd-4eb1-bf65-a0c6a82cb956 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a606ddb7c1b346889111a569520789a9', 'request_id': 'req-a2bed543-c5dd-4eb1-bf65-a0c6a82cb956'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:42.764940 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a2bed543-c5dd-4eb1-bf65-a0c6a82cb956 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a606ddb7c1b346889111a569520789a9', 'request_id': 'req-a2bed543-c5dd-4eb1-bf65-a0c6a82cb956'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:42.779851 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a2bed543-c5dd-4eb1-bf65-a0c6a82cb956 admin admin] The token cache is being invalidate because project a606ddb7c1b346889111a569520789a9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81405) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:27:42.780283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a2bed543-c5dd-4eb1-bf65-a0c6a82cb956 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:42.780782 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-a2bed543-c5dd-4eb1-bf65-a0c6a82cb956 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:27:42.782546 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1454/2896] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:27:42 2022] DELETE /identity/v3/projects/a606ddb7c1b346889111a569520789a9 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:27:42.787907 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94b797c-9068-4ba6-a579-765fed977931 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:42.788310 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94b797c-9068-4ba6-a579-765fed977931 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:42.788765 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94b797c-9068-4ba6-a579-765fed977931 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:42.810072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-b94b797c-9068-4ba6-a579-765fed977931 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:42.811479 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-b94b797c-9068-4ba6-a579-765fed977931 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:42.822142 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:42.882636 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1443/2897] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:42.890457 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807e19dd-dda9-4c82-a1a6-85ce7a4f5306 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:42.890815 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807e19dd-dda9-4c82-a1a6-85ce7a4f5306 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:42.891122 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807e19dd-dda9-4c82-a1a6-85ce7a4f5306 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:42.913751 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-807e19dd-dda9-4c82-a1a6-85ce7a4f5306 None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:42.914884 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-807e19dd-dda9-4c82-a1a6-85ce7a4f5306 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:42.922903 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1455/2898] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:42.927342 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94b797c-9068-4ba6-a579-765fed977931 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:42.937794 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94b797c-9068-4ba6-a579-765fed977931 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:42.943587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588ad2fa-beb2-4b29-b5a4-dadf2b342bb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:42.944688 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588ad2fa-beb2-4b29-b5a4-dadf2b342bb9 admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:42.944973 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588ad2fa-beb2-4b29-b5a4-dadf2b342bb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:42.945246 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588ad2fa-beb2-4b29-b5a4-dadf2b342bb9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:42.945893 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588ad2fa-beb2-4b29-b5a4-dadf2b342bb9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81404) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 01:27:42.946195 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588ad2fa-beb2-4b29-b5a4-dadf2b342bb9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:42.947837 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588ad2fa-beb2-4b29-b5a4-dadf2b342bb9 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:42.963100 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1444/2899] 10.222.0.15 () {58 vars in 1153 bytes} [Tue Sep 13 01:27:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 01:27:42.967807 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807e19dd-dda9-4c82-a1a6-85ce7a4f5306 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:42.980412 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807e19dd-dda9-4c82-a1a6-85ce7a4f5306 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:42.985410 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93369fe2-32a2-4a7b-a1b4-f05f7f24e289 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:42.986170 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93369fe2-32a2-4a7b-a1b4-f05f7f24e289 admin admin] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:42.986458 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93369fe2-32a2-4a7b-a1b4-f05f7f24e289 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:42.986735 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93369fe2-32a2-4a7b-a1b4-f05f7f24e289 admin admin] PATH_INFO: `/v3/projects` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:42.988174 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93369fe2-32a2-4a7b-a1b4-f05f7f24e289 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:42.995138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93369fe2-32a2-4a7b-a1b4-f05f7f24e289 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:43.022494 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1456/2900] 10.222.0.15 () {60 vars in 1153 bytes} [Tue Sep 13 01:27:42 2022] POST /identity/v3/projects => generated 381 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 01:27:43.026784 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588ad2fa-beb2-4b29-b5a4-dadf2b342bb9 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:43.039842 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588ad2fa-beb2-4b29-b5a4-dadf2b342bb9 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:43.108242 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513c04d7-f83a-49a0-9808-3d0f059ae859 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:43.108972 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513c04d7-f83a-49a0-9808-3d0f059ae859 admin admin] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:43.109333 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513c04d7-f83a-49a0-9808-3d0f059ae859 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:43.109614 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513c04d7-f83a-49a0-9808-3d0f059ae859 admin admin] PATH_INFO: `/v3/users` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:43.110274 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-513c04d7-f83a-49a0-9808-3d0f059ae859 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:43.116679 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-513c04d7-f83a-49a0-9808-3d0f059ae859 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:43.141568 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1445/2901] 10.222.0.15 () {60 vars in 1144 bytes} [Tue Sep 13 01:27:43 2022] POST /identity/v3/users => generated 291 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 01:27:43.146809 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93369fe2-32a2-4a7b-a1b4-f05f7f24e289 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:43.157950 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93369fe2-32a2-4a7b-a1b4-f05f7f24e289 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:43.163298 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc804d3-674d-4e83-b39f-486a1f66ec5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:43.163977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc804d3-674d-4e83-b39f-486a1f66ec5f admin admin] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:43.164202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc804d3-674d-4e83-b39f-486a1f66ec5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:43.164480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc804d3-674d-4e83-b39f-486a1f66ec5f admin admin] PATH_INFO: `/v3/roles` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:43.165124 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc804d3-674d-4e83-b39f-486a1f66ec5f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:43.166917 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc804d3-674d-4e83-b39f-486a1f66ec5f admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:43.178388 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1457/2902] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:27:43 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 01:27:43.182579 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513c04d7-f83a-49a0-9808-3d0f059ae859 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:43.193359 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513c04d7-f83a-49a0-9808-3d0f059ae859 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:43.198111 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d1f158-5b9e-4cb3-b670-4784b0fcf20b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:43.198785 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d1f158-5b9e-4cb3-b670-4784b0fcf20b admin admin] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:43.199172 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d1f158-5b9e-4cb3-b670-4784b0fcf20b admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:43.199446 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d1f158-5b9e-4cb3-b670-4784b0fcf20b admin admin] PATH_INFO: `/v3/roles` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:43.200078 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d1f158-5b9e-4cb3-b670-4784b0fcf20b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:43.201739 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d1f158-5b9e-4cb3-b670-4784b0fcf20b admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:43.213607 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1446/2903] 10.222.0.15 () {58 vars in 1122 bytes} [Tue Sep 13 01:27:43 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 01:27:43.218036 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc804d3-674d-4e83-b39f-486a1f66ec5f admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:43.229590 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc804d3-674d-4e83-b39f-486a1f66ec5f admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:43.234642 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4ac7e5-597f-481c-8e2e-f1dca1e937c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:43.235410 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4ac7e5-597f-481c-8e2e-f1dca1e937c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:43.235823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4ac7e5-597f-481c-8e2e-f1dca1e937c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:43.236162 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4ac7e5-597f-481c-8e2e-f1dca1e937c6 admin admin] PATH_INFO: `/v3/projects/53b74c0569e9452d817797f3f9d3a78f/users/ed67fb86412a404d9acd882d88432b8c/roles/6f71034643cb4e8caee299f5d0bb863f` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:43.253900 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4ac7e5-597f-481c-8e2e-f1dca1e937c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=53b74c0569e9452d817797f3f9d3a78f, user_id=ed67fb86412a404d9acd882d88432b8c, role_id=6f71034643cb4e8caee299f5d0bb863f)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:43.257196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4ac7e5-597f-481c-8e2e-f1dca1e937c6 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:43.272321 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1458/2904] 10.222.0.15 () {60 vars in 1483 bytes} [Tue Sep 13 01:27:43 2022] PUT /identity/v3/projects/53b74c0569e9452d817797f3f9d3a78f/users/ed67fb86412a404d9acd882d88432b8c/roles/6f71034643cb4e8caee299f5d0bb863f => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 01:27:43.277411 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ab2f82-fe5f-4f05-a272-c2a696e305fa None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:43.278148 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ab2f82-fe5f-4f05-a272-c2a696e305fa None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:43.278595 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ab2f82-fe5f-4f05-a272-c2a696e305fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:43.301699 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-79ab2f82-fe5f-4f05-a272-c2a696e305fa None None] MFA Rules not processed for user `22ddb5151c0f4696a4f99a9acc8804ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:43.303616 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-79ab2f82-fe5f-4f05-a272-c2a696e305fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:43.388427 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1447/2905] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 01:27:43.399277 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4ac7e5-597f-481c-8e2e-f1dca1e937c6 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:43.401099 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-7b4ac7e5-597f-481c-8e2e-f1dca1e937c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:43.413645 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:43.483090 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4ac7e5-597f-481c-8e2e-f1dca1e937c6 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:43.487514 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776a32e4-8565-482e-8e26-30a8dd9dd8cf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:43.488389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776a32e4-8565-482e-8e26-30a8dd9dd8cf service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:43.488710 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776a32e4-8565-482e-8e26-30a8dd9dd8cf service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:43.498486 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776a32e4-8565-482e-8e26-30a8dd9dd8cf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:43.517318 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776a32e4-8565-482e-8e26-30a8dd9dd8cf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:43.518789 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776a32e4-8565-482e-8e26-30a8dd9dd8cf service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:43.537689 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1459/2906] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:27:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:27:44.279378 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9eaebf-d1c8-4586-aaf3-887e564e5093 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:44.280560 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9eaebf-d1c8-4586-aaf3-887e564e5093 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:44.280987 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9eaebf-d1c8-4586-aaf3-887e564e5093 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:44.318011 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-df9eaebf-d1c8-4586-aaf3-887e564e5093 None None] MFA Rules not processed for user `ed67fb86412a404d9acd882d88432b8c`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:44.319585 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-df9eaebf-d1c8-4586-aaf3-887e564e5093 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:44.339190 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:44.466358 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1448/2907] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:44 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:44.477451 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776a32e4-8565-482e-8e26-30a8dd9dd8cf service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:27:44.481809 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-776a32e4-8565-482e-8e26-30a8dd9dd8cf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:44.553434 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776a32e4-8565-482e-8e26-30a8dd9dd8cf service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:27:44.558225 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73de8c11-726e-48c7-a80e-415e2b77552a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:27:44.559075 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73de8c11-726e-48c7-a80e-415e2b77552a service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:44.559417 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73de8c11-726e-48c7-a80e-415e2b77552a service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:44.559749 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73de8c11-726e-48c7-a80e-415e2b77552a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:44.569369 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73de8c11-726e-48c7-a80e-415e2b77552a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:27:44.570945 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73de8c11-726e-48c7-a80e-415e2b77552a service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:27:44.582565 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1460/2908] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:27:44 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:27:46.075961 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd1d4a3-4966-4387-92e4-c90998d25956 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:27:46.076587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd1d4a3-4966-4387-92e4-c90998d25956 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:27:46.076587 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd1d4a3-4966-4387-92e4-c90998d25956 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:27:46.101335 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-afd1d4a3-4966-4387-92e4-c90998d25956 None None] MFA Rules not processed for user `ed67fb86412a404d9acd882d88432b8c`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:27:46.102567 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-afd1d4a3-4966-4387-92e4-c90998d25956 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:27:46.110241 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1449/2909] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:27:46 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:27:54.576702 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=51 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:27:56.091263 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=25 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:28:15.190202 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73de8c11-726e-48c7-a80e-415e2b77552a service cinder] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:28:15.193764 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-73de8c11-726e-48c7-a80e-415e2b77552a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:28:15.215143 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73de8c11-726e-48c7-a80e-415e2b77552a service cinder] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:28:15.220393 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c313faa5-b53b-40cd-9db1-6820760dc027 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:28:15.221439 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c313faa5-b53b-40cd-9db1-6820760dc027 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:28:15.221936 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c313faa5-b53b-40cd-9db1-6820760dc027 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:28:15.222368 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c313faa5-b53b-40cd-9db1-6820760dc027 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:28:15.224918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c313faa5-b53b-40cd-9db1-6820760dc027 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:28:15.314283 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c313faa5-b53b-40cd-9db1-6820760dc027 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:28:15.315823 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c313faa5-b53b-40cd-9db1-6820760dc027 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:28:15.327976 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1461/2910] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:28:15 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:28:25.321547 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=30 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:28:29.697334 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd1d4a3-4966-4387-92e4-c90998d25956 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:28:29.706957 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd1d4a3-4966-4387-92e4-c90998d25956 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:28:29.710448 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55647f22-f5e9-4106-b1ca-e0f0331d3253 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:28:29.711090 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55647f22-f5e9-4106-b1ca-e0f0331d3253 service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:28:29.711346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55647f22-f5e9-4106-b1ca-e0f0331d3253 service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:28:29.711612 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55647f22-f5e9-4106-b1ca-e0f0331d3253 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:28:29.713332 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-55647f22-f5e9-4106-b1ca-e0f0331d3253 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:28:29.724786 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55647f22-f5e9-4106-b1ca-e0f0331d3253 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:28:29.726198 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55647f22-f5e9-4106-b1ca-e0f0331d3253 service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:28:29.735709 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1450/2911] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:28:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:28:35.925001 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c313faa5-b53b-40cd-9db1-6820760dc027 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:28:35.926730 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-c313faa5-b53b-40cd-9db1-6820760dc027 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:28:35.951135 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c313faa5-b53b-40cd-9db1-6820760dc027 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:28:35.958138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec64ebe-e284-4946-8419-e1bdd186611b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:28:35.959138 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec64ebe-e284-4946-8419-e1bdd186611b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:28:35.959454 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec64ebe-e284-4946-8419-e1bdd186611b admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:28:35.959742 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec64ebe-e284-4946-8419-e1bdd186611b admin admin] PATH_INFO: `/v3/users/2842b222fe6b4821a670d13ec8febf1a` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:28:35.961424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec64ebe-e284-4946-8419-e1bdd186611b admin admin] RBAC: Authorizing `identity:delete_user(user_id=2842b222fe6b4821a670d13ec8febf1a)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:28:35.963384 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec64ebe-e284-4946-8419-e1bdd186611b admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:28:36.014183 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0ec64ebe-e284-4946-8419-e1bdd186611b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2842b222fe6b4821a670d13ec8febf1a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.014838 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0ec64ebe-e284-4946-8419-e1bdd186611b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2842b222fe6b4821a670d13ec8febf1a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.024112 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0ec64ebe-e284-4946-8419-e1bdd186611b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2842b222fe6b4821a670d13ec8febf1a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.024739 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0ec64ebe-e284-4946-8419-e1bdd186611b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2842b222fe6b4821a670d13ec8febf1a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.037345 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-0ec64ebe-e284-4946-8419-e1bdd186611b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2842b222fe6b4821a670d13ec8febf1a'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.049995 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1462/2912] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:28:35 2022] DELETE /identity/v3/users/2842b222fe6b4821a670d13ec8febf1a => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:28:36.220070 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55647f22-f5e9-4106-b1ca-e0f0331d3253 service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:28:36.230517 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55647f22-f5e9-4106-b1ca-e0f0331d3253 service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:28:36.288095 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48a7800-5d66-41e3-b749-06ec1a79e178 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:28:36.288717 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48a7800-5d66-41e3-b749-06ec1a79e178 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:28:36.288988 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48a7800-5d66-41e3-b749-06ec1a79e178 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:28:36.289250 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48a7800-5d66-41e3-b749-06ec1a79e178 admin admin] PATH_INFO: `/v3/projects/66cdd4ec03ea4fbf959ede65bb96f6ca` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:28:36.290353 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48a7800-5d66-41e3-b749-06ec1a79e178 admin admin] RBAC: Authorizing `identity:delete_project(project_id=66cdd4ec03ea4fbf959ede65bb96f6ca)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:28:36.292259 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48a7800-5d66-41e3-b749-06ec1a79e178 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:28:36.337249 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f48a7800-5d66-41e3-b749-06ec1a79e178 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '66cdd4ec03ea4fbf959ede65bb96f6ca', 'request_id': 'req-f48a7800-5d66-41e3-b749-06ec1a79e178'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.342461 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f48a7800-5d66-41e3-b749-06ec1a79e178 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '66cdd4ec03ea4fbf959ede65bb96f6ca', 'request_id': 'req-f48a7800-5d66-41e3-b749-06ec1a79e178'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.348438 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f48a7800-5d66-41e3-b749-06ec1a79e178 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '66cdd4ec03ea4fbf959ede65bb96f6ca', 'request_id': 'req-f48a7800-5d66-41e3-b749-06ec1a79e178'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.359864 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f48a7800-5d66-41e3-b749-06ec1a79e178 admin admin] The token cache is being invalidate because project 66cdd4ec03ea4fbf959ede65bb96f6ca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:28:36.360147 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f48a7800-5d66-41e3-b749-06ec1a79e178 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.360758 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-f48a7800-5d66-41e3-b749-06ec1a79e178 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.362198 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1451/2913] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:28:36 2022] DELETE /identity/v3/projects/66cdd4ec03ea4fbf959ede65bb96f6ca => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:28:36.366742 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec64ebe-e284-4946-8419-e1bdd186611b admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:28:36.368212 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-0ec64ebe-e284-4946-8419-e1bdd186611b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:28:36.382184 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec64ebe-e284-4946-8419-e1bdd186611b admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:28:36.438201 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c6495b-9ecf-4be5-9592-d70e1739169d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:28:36.438760 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c6495b-9ecf-4be5-9592-d70e1739169d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:28:36.439063 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c6495b-9ecf-4be5-9592-d70e1739169d admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:28:36.439333 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c6495b-9ecf-4be5-9592-d70e1739169d admin admin] PATH_INFO: `/v3/users/0d17e633508f4d0eb77a81bb840ce7b9` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:28:36.440444 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c6495b-9ecf-4be5-9592-d70e1739169d admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d17e633508f4d0eb77a81bb840ce7b9)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:28:36.442992 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c6495b-9ecf-4be5-9592-d70e1739169d admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:28:36.488118 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e4c6495b-9ecf-4be5-9592-d70e1739169d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d17e633508f4d0eb77a81bb840ce7b9'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.488755 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e4c6495b-9ecf-4be5-9592-d70e1739169d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d17e633508f4d0eb77a81bb840ce7b9'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.495858 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e4c6495b-9ecf-4be5-9592-d70e1739169d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d17e633508f4d0eb77a81bb840ce7b9'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.496664 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e4c6495b-9ecf-4be5-9592-d70e1739169d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d17e633508f4d0eb77a81bb840ce7b9'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.509222 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-e4c6495b-9ecf-4be5-9592-d70e1739169d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d17e633508f4d0eb77a81bb840ce7b9'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.511226 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:28:36.523767 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1463/2914] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:28:36 2022] DELETE /identity/v3/users/0d17e633508f4d0eb77a81bb840ce7b9 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:28:36.693789 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48a7800-5d66-41e3-b749-06ec1a79e178 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:28:36.695346 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-f48a7800-5d66-41e3-b749-06ec1a79e178 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:28:36.708458 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48a7800-5d66-41e3-b749-06ec1a79e178 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:28:36.765041 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb98ed97-0e7d-4d09-83e0-5963cecb9647 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:28:36.765666 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb98ed97-0e7d-4d09-83e0-5963cecb9647 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:28:36.765943 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb98ed97-0e7d-4d09-83e0-5963cecb9647 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:28:36.766209 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb98ed97-0e7d-4d09-83e0-5963cecb9647 admin admin] PATH_INFO: `/v3/projects/01cdb885e3894f3eb1d6947137d35b67` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:28:36.767394 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb98ed97-0e7d-4d09-83e0-5963cecb9647 admin admin] RBAC: Authorizing `identity:delete_project(project_id=01cdb885e3894f3eb1d6947137d35b67)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:28:36.769308 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb98ed97-0e7d-4d09-83e0-5963cecb9647 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:28:36.790769 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:28:36.816400 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-fb98ed97-0e7d-4d09-83e0-5963cecb9647 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '01cdb885e3894f3eb1d6947137d35b67', 'request_id': 'req-fb98ed97-0e7d-4d09-83e0-5963cecb9647'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.821830 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-fb98ed97-0e7d-4d09-83e0-5963cecb9647 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '01cdb885e3894f3eb1d6947137d35b67', 'request_id': 'req-fb98ed97-0e7d-4d09-83e0-5963cecb9647'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.829815 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-fb98ed97-0e7d-4d09-83e0-5963cecb9647 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '01cdb885e3894f3eb1d6947137d35b67', 'request_id': 'req-fb98ed97-0e7d-4d09-83e0-5963cecb9647'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.843112 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-fb98ed97-0e7d-4d09-83e0-5963cecb9647 admin admin] The token cache is being invalidate because project 01cdb885e3894f3eb1d6947137d35b67 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:28:36.843401 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-fb98ed97-0e7d-4d09-83e0-5963cecb9647 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.844016 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-fb98ed97-0e7d-4d09-83e0-5963cecb9647 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:28:36.845667 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1452/2915] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:28:36 2022] DELETE /identity/v3/projects/01cdb885e3894f3eb1d6947137d35b67 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:28:38.010356 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c6495b-9ecf-4be5-9592-d70e1739169d admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:28:38.011716 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-e4c6495b-9ecf-4be5-9592-d70e1739169d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:28:38.067955 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c6495b-9ecf-4be5-9592-d70e1739169d admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:28:38.114926 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84012dd4-c6cb-4553-a357-5e2b60c63abe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f90a5e57daa643eab94c01e21fcbc4c4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:28:38.115531 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84012dd4-c6cb-4553-a357-5e2b60c63abe service cinder] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:28:38.115808 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84012dd4-c6cb-4553-a357-5e2b60c63abe service cinder] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:28:38.116081 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84012dd4-c6cb-4553-a357-5e2b60c63abe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:28:38.117771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-84012dd4-c6cb-4553-a357-5e2b60c63abe service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:28:38.203021 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84012dd4-c6cb-4553-a357-5e2b60c63abe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:28:38.204493 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84012dd4-c6cb-4553-a357-5e2b60c63abe service cinder] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:28:38.225096 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1464/2916] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:28:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:28:40.816695 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb98ed97-0e7d-4d09-83e0-5963cecb9647 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:28:40.818185 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-fb98ed97-0e7d-4d09-83e0-5963cecb9647 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:28:40.874089 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb98ed97-0e7d-4d09-83e0-5963cecb9647 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:28:40.877643 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fc59e4-15e1-4fc1-93c7-e91b2363f93c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:28:40.878906 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fc59e4-15e1-4fc1-93c7-e91b2363f93c service neutron] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:28:40.879187 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fc59e4-15e1-4fc1-93c7-e91b2363f93c service neutron] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:28:40.879456 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fc59e4-15e1-4fc1-93c7-e91b2363f93c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:28:40.881135 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-65fc59e4-15e1-4fc1-93c7-e91b2363f93c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:28:40.894161 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65fc59e4-15e1-4fc1-93c7-e91b2363f93c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:28:40.895609 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65fc59e4-15e1-4fc1-93c7-e91b2363f93c service neutron] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:28:40.906460 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1453/2917] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:28:40 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:28:48.222598 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=87 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:28:50.901473 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=42,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:28:50.928139 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b9ce38-5fce-49c0-b251-c8654723b1f9 None None] REQUEST_METHOD: `POST` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:28:50.928657 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b9ce38-5fce-49c0-b251-c8654723b1f9 None None] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:28:50.928765 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b9ce38-5fce-49c0-b251-c8654723b1f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:28:50.947916 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.auth.core [None req-86b9ce38-5fce-49c0-b251-c8654723b1f9 None None] MFA Rules not processed for user `ed67fb86412a404d9acd882d88432b8c`. Rule list: `[]` (Enabled: `True`). {{(pid=81405) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:28:50.949087 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-86b9ce38-5fce-49c0-b251-c8654723b1f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:28:51.007567 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1465/2918] 10.222.0.15 () {58 vars in 958 bytes} [Tue Sep 13 01:28:50 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:29:01.000290 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=25 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:29:36.506053 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fc59e4-15e1-4fc1-93c7-e91b2363f93c service neutron] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:29:36.508174 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-65fc59e4-15e1-4fc1-93c7-e91b2363f93c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:29:36.529027 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fc59e4-15e1-4fc1-93c7-e91b2363f93c service neutron] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:29:36.551214 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb7eac9-20eb-453b-96cc-33d80d674d95 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f2da4d2d3604d16b58dc25b8c17c49a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:29:36.552812 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb7eac9-20eb-453b-96cc-33d80d674d95 service nova] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:29:36.553446 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb7eac9-20eb-453b-96cc-33d80d674d95 service nova] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:29:36.553962 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb7eac9-20eb-453b-96cc-33d80d674d95 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:29:36.556112 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-ceb7eac9-20eb-453b-96cc-33d80d674d95 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:29:36.573205 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb7eac9-20eb-453b-96cc-33d80d674d95 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:29:36.579194 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb7eac9-20eb-453b-96cc-33d80d674d95 service nova] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:29:36.591087 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1454/2919] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:29:36 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:29:36.665832 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b9ce38-5fce-49c0-b251-c8654723b1f9 None None] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:29:36.667237 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-86b9ce38-5fce-49c0-b251-c8654723b1f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:29:36.682144 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b9ce38-5fce-49c0-b251-c8654723b1f9 None None] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:29:36.686776 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275c432b-0515-49e8-a87d-56044661b3c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '09cd263f72484733adaaa0850ebfa609', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:29:36.687864 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275c432b-0515-49e8-a87d-56044661b3c4 service neutron] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:29:36.688063 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275c432b-0515-49e8-a87d-56044661b3c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:29:36.688436 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275c432b-0515-49e8-a87d-56044661b3c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:29:36.690678 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-275c432b-0515-49e8-a87d-56044661b3c4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:29:36.702022 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275c432b-0515-49e8-a87d-56044661b3c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:29:36.703937 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275c432b-0515-49e8-a87d-56044661b3c4 service neutron] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:29:36.714879 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1466/2920] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:29:36 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:29:43.877048 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc8322b-a302-424d-9981-ae73169508c5 None None] REQUEST_METHOD: `POST` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:29:43.877523 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc8322b-a302-424d-9981-ae73169508c5 None None] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:29:43.877633 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc8322b-a302-424d-9981-ae73169508c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:29:43.927364 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.auth.core [None req-1bc8322b-a302-424d-9981-ae73169508c5 None None] MFA Rules not processed for user `e28a0ba45389417ba8ae92376dba87ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81404) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 01:29:43.928571 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1bc8322b-a302-424d-9981-ae73169508c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:29:43.993261 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1455/2921] 10.222.0.15 () {58 vars in 967 bytes} [Tue Sep 13 01:29:43 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 01:29:44.033717 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275c432b-0515-49e8-a87d-56044661b3c4 service neutron] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:29:44.035068 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-275c432b-0515-49e8-a87d-56044661b3c4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:29:44.126249 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275c432b-0515-49e8-a87d-56044661b3c4 service neutron] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:29:44.131809 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1033cb-220b-4eb1-a1b4-db2b2ef239ef service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0d7d8ed4ede4ddca192fce0a497689d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:29:44.132411 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1033cb-220b-4eb1-a1b4-db2b2ef239ef service swift] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:29:44.132691 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1033cb-220b-4eb1-a1b4-db2b2ef239ef service swift] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:29:44.132962 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1033cb-220b-4eb1-a1b4-db2b2ef239ef service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:29:44.140857 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1033cb-220b-4eb1-a1b4-db2b2ef239ef service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:29:44.142417 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1033cb-220b-4eb1-a1b4-db2b2ef239ef service swift] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:29:44.151609 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1467/2922] 10.222.0.15 () {60 vars in 1384 bytes} [Tue Sep 13 01:29:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 01:29:46.466367 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc8322b-a302-424d-9981-ae73169508c5 None None] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:29:46.468387 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-1bc8322b-a302-424d-9981-ae73169508c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:29:46.548674 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc8322b-a302-424d-9981-ae73169508c5 None None] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:29:46.554675 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332ab45d-8ab6-4d68-91ad-29e17f7dd617 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8eaff1b843a145a49994dcf089f44315', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:29:46.555386 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332ab45d-8ab6-4d68-91ad-29e17f7dd617 service placement] REQUEST_METHOD: `GET` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:29:46.555737 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332ab45d-8ab6-4d68-91ad-29e17f7dd617 service placement] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:29:46.556237 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332ab45d-8ab6-4d68-91ad-29e17f7dd617 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:29:46.558024 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-332ab45d-8ab6-4d68-91ad-29e17f7dd617 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:29:46.577600 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332ab45d-8ab6-4d68-91ad-29e17f7dd617 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:29:46.579280 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332ab45d-8ab6-4d68-91ad-29e17f7dd617 service placement] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:29:46.580125 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1033cb-220b-4eb1-a1b4-db2b2ef239ef service swift] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:29:46.583045 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-9b1033cb-220b-4eb1-a1b4-db2b2ef239ef service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:29:46.592101 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1456/2923] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:29:46 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:29:46.691278 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1033cb-220b-4eb1-a1b4-db2b2ef239ef service swift] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:29:46.695766 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ed6823-f63b-4980-b75e-db1990024eee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:29:46.696389 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ed6823-f63b-4980-b75e-db1990024eee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:29:46.696665 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ed6823-f63b-4980-b75e-db1990024eee admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:29:46.696933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ed6823-f63b-4980-b75e-db1990024eee admin admin] PATH_INFO: `/v3/users/ed67fb86412a404d9acd882d88432b8c` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:29:46.698044 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ed6823-f63b-4980-b75e-db1990024eee admin admin] RBAC: Authorizing `identity:delete_user(user_id=ed67fb86412a404d9acd882d88432b8c)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:29:46.699977 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ed6823-f63b-4980-b75e-db1990024eee admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:29:46.746707 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b3ed6823-f63b-4980-b75e-db1990024eee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ed67fb86412a404d9acd882d88432b8c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:46.747424 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b3ed6823-f63b-4980-b75e-db1990024eee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ed67fb86412a404d9acd882d88432b8c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:46.754933 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b3ed6823-f63b-4980-b75e-db1990024eee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ed67fb86412a404d9acd882d88432b8c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:46.755579 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b3ed6823-f63b-4980-b75e-db1990024eee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ed67fb86412a404d9acd882d88432b8c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:46.760121 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:29:46.771771 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-b3ed6823-f63b-4980-b75e-db1990024eee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ed67fb86412a404d9acd882d88432b8c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:46.784439 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1468/2924] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:29:46 2022] DELETE /identity/v3/users/ed67fb86412a404d9acd882d88432b8c => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:29:46.962210 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332ab45d-8ab6-4d68-91ad-29e17f7dd617 service placement] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:29:46.998347 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332ab45d-8ab6-4d68-91ad-29e17f7dd617 service placement] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:29:47.055581 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=100 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:29:47.074320 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e630f86-01d8-4e21-8f3d-1b9f913bfaac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:29:47.075072 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e630f86-01d8-4e21-8f3d-1b9f913bfaac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:29:47.075422 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e630f86-01d8-4e21-8f3d-1b9f913bfaac admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:29:47.075765 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e630f86-01d8-4e21-8f3d-1b9f913bfaac admin admin] PATH_INFO: `/v3/projects/53b74c0569e9452d817797f3f9d3a78f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:29:47.076970 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e630f86-01d8-4e21-8f3d-1b9f913bfaac admin admin] RBAC: Authorizing `identity:delete_project(project_id=53b74c0569e9452d817797f3f9d3a78f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:29:47.078892 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e630f86-01d8-4e21-8f3d-1b9f913bfaac admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:29:47.138900 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4e630f86-01d8-4e21-8f3d-1b9f913bfaac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '53b74c0569e9452d817797f3f9d3a78f', 'request_id': 'req-4e630f86-01d8-4e21-8f3d-1b9f913bfaac'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:47.145986 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4e630f86-01d8-4e21-8f3d-1b9f913bfaac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '53b74c0569e9452d817797f3f9d3a78f', 'request_id': 'req-4e630f86-01d8-4e21-8f3d-1b9f913bfaac'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:47.152545 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4e630f86-01d8-4e21-8f3d-1b9f913bfaac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '53b74c0569e9452d817797f3f9d3a78f', 'request_id': 'req-4e630f86-01d8-4e21-8f3d-1b9f913bfaac'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:47.165140 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4e630f86-01d8-4e21-8f3d-1b9f913bfaac admin admin] The token cache is being invalidate because project 53b74c0569e9452d817797f3f9d3a78f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:29:47.165566 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4e630f86-01d8-4e21-8f3d-1b9f913bfaac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:47.166199 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-4e630f86-01d8-4e21-8f3d-1b9f913bfaac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:47.167745 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1457/2925] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:29:46 2022] DELETE /identity/v3/projects/53b74c0569e9452d817797f3f9d3a78f => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:29:51.024573 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ed6823-f63b-4980-b75e-db1990024eee admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:29:51.026120 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-b3ed6823-f63b-4980-b75e-db1990024eee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:29:51.039311 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ed6823-f63b-4980-b75e-db1990024eee admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:29:51.100276 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6991d858-9217-471e-96c6-c4eb49fe63e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:29:51.101196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6991d858-9217-471e-96c6-c4eb49fe63e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:29:51.101480 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6991d858-9217-471e-96c6-c4eb49fe63e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:29:51.101779 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6991d858-9217-471e-96c6-c4eb49fe63e3 admin admin] PATH_INFO: `/v3/users/d54805410115495597ef0720af2acf22` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:29:51.103210 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6991d858-9217-471e-96c6-c4eb49fe63e3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d54805410115495597ef0720af2acf22)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:29:51.105227 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6991d858-9217-471e-96c6-c4eb49fe63e3 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:29:51.156581 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6991d858-9217-471e-96c6-c4eb49fe63e3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd54805410115495597ef0720af2acf22'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:51.157233 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6991d858-9217-471e-96c6-c4eb49fe63e3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd54805410115495597ef0720af2acf22'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:51.164613 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6991d858-9217-471e-96c6-c4eb49fe63e3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd54805410115495597ef0720af2acf22'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:51.165289 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6991d858-9217-471e-96c6-c4eb49fe63e3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd54805410115495597ef0720af2acf22'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:51.177560 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-6991d858-9217-471e-96c6-c4eb49fe63e3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd54805410115495597ef0720af2acf22'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:51.190126 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1469/2926] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:29:51 2022] DELETE /identity/v3/users/d54805410115495597ef0720af2acf22 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:29:51.355470 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e630f86-01d8-4e21-8f3d-1b9f913bfaac admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:29:51.357132 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-4e630f86-01d8-4e21-8f3d-1b9f913bfaac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:29:51.370405 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e630f86-01d8-4e21-8f3d-1b9f913bfaac admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:29:51.428064 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b31769-c58f-40a3-8dfd-0b88afb3ea52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:29:51.428698 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b31769-c58f-40a3-8dfd-0b88afb3ea52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:29:51.428971 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b31769-c58f-40a3-8dfd-0b88afb3ea52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:29:51.429238 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b31769-c58f-40a3-8dfd-0b88afb3ea52 admin admin] PATH_INFO: `/v3/projects/7bede684fde243038bcb6b0d5c31ac8f` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:29:51.430338 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b31769-c58f-40a3-8dfd-0b88afb3ea52 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7bede684fde243038bcb6b0d5c31ac8f)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:29:51.432261 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b31769-c58f-40a3-8dfd-0b88afb3ea52 admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:29:51.480850 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-16b31769-c58f-40a3-8dfd-0b88afb3ea52 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7bede684fde243038bcb6b0d5c31ac8f', 'request_id': 'req-16b31769-c58f-40a3-8dfd-0b88afb3ea52'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:51.486241 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-16b31769-c58f-40a3-8dfd-0b88afb3ea52 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7bede684fde243038bcb6b0d5c31ac8f', 'request_id': 'req-16b31769-c58f-40a3-8dfd-0b88afb3ea52'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:51.492260 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-16b31769-c58f-40a3-8dfd-0b88afb3ea52 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7bede684fde243038bcb6b0d5c31ac8f', 'request_id': 'req-16b31769-c58f-40a3-8dfd-0b88afb3ea52'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:51.505553 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-16b31769-c58f-40a3-8dfd-0b88afb3ea52 admin admin] The token cache is being invalidate because project 7bede684fde243038bcb6b0d5c31ac8f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:29:51.505838 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-16b31769-c58f-40a3-8dfd-0b88afb3ea52 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:51.506452 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-16b31769-c58f-40a3-8dfd-0b88afb3ea52 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:51.507940 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1458/2927] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:29:51 2022] DELETE /identity/v3/projects/7bede684fde243038bcb6b0d5c31ac8f => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:29:51.512034 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6991d858-9217-471e-96c6-c4eb49fe63e3 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:29:51.513435 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-6991d858-9217-471e-96c6-c4eb49fe63e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:29:51.526969 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6991d858-9217-471e-96c6-c4eb49fe63e3 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:29:51.576054 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:29:51.584463 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8054c32-3b4a-4417-be8c-51f17dcdf0a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:29:51.585157 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8054c32-3b4a-4417-be8c-51f17dcdf0a4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:29:51.585544 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8054c32-3b4a-4417-be8c-51f17dcdf0a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:29:51.585860 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8054c32-3b4a-4417-be8c-51f17dcdf0a4 admin admin] PATH_INFO: `/v3/users/7b1e39df9fd8471f9f36e52e2a1f7f0c` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:29:51.587238 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8054c32-3b4a-4417-be8c-51f17dcdf0a4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7b1e39df9fd8471f9f36e52e2a1f7f0c)` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:29:51.589353 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8054c32-3b4a-4417-be8c-51f17dcdf0a4 admin admin] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:29:51.636197 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f8054c32-3b4a-4417-be8c-51f17dcdf0a4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7b1e39df9fd8471f9f36e52e2a1f7f0c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:51.636861 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f8054c32-3b4a-4417-be8c-51f17dcdf0a4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7b1e39df9fd8471f9f36e52e2a1f7f0c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:51.644196 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f8054c32-3b4a-4417-be8c-51f17dcdf0a4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7b1e39df9fd8471f9f36e52e2a1f7f0c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:51.645449 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f8054c32-3b4a-4417-be8c-51f17dcdf0a4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7b1e39df9fd8471f9f36e52e2a1f7f0c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:51.657703 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.notifications [None req-f8054c32-3b4a-4417-be8c-51f17dcdf0a4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7b1e39df9fd8471f9f36e52e2a1f7f0c'} {{(pid=81405) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:51.669684 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1470/2928] 10.222.0.15 () {58 vars in 1224 bytes} [Tue Sep 13 01:29:51 2022] DELETE /identity/v3/users/7b1e39df9fd8471f9f36e52e2a1f7f0c => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:29:51.841649 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b31769-c58f-40a3-8dfd-0b88afb3ea52 admin admin] Authenticating user token {{(pid=81404) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:29:51.843457 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.fernet_utils [None req-16b31769-c58f-40a3-8dfd-0b88afb3ea52 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81404) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:29:51.870297 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b31769-c58f-40a3-8dfd-0b88afb3ea52 admin admin] Validating token access rules against request {{(pid=81404) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:29:51.874998 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:29:51.931166 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abc1f5c-aa9a-4f04-8670-afe09d2a93aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '22ddb5151c0f4696a4f99a9acc8804ea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ce37ba2e4948471599c41255ecf54622', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81404) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:29:51.931813 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abc1f5c-aa9a-4f04-8670-afe09d2a93aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:29:51.932096 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abc1f5c-aa9a-4f04-8670-afe09d2a93aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:29:51.932373 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abc1f5c-aa9a-4f04-8670-afe09d2a93aa admin admin] PATH_INFO: `/v3/projects/bcbe9f2e8f6a4faeb418d1bbc59a47dc` {{(pid=81404) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:29:51.933528 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abc1f5c-aa9a-4f04-8670-afe09d2a93aa admin admin] RBAC: Authorizing `identity:delete_project(project_id=bcbe9f2e8f6a4faeb418d1bbc59a47dc)` {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:29:51.935485 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abc1f5c-aa9a-4f04-8670-afe09d2a93aa admin admin] RBAC: Authorization granted {{(pid=81404) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:29:51.984724 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6abc1f5c-aa9a-4f04-8670-afe09d2a93aa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bcbe9f2e8f6a4faeb418d1bbc59a47dc', 'request_id': 'req-6abc1f5c-aa9a-4f04-8670-afe09d2a93aa'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:51.991004 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6abc1f5c-aa9a-4f04-8670-afe09d2a93aa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bcbe9f2e8f6a4faeb418d1bbc59a47dc', 'request_id': 'req-6abc1f5c-aa9a-4f04-8670-afe09d2a93aa'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:52.000710 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6abc1f5c-aa9a-4f04-8670-afe09d2a93aa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bcbe9f2e8f6a4faeb418d1bbc59a47dc', 'request_id': 'req-6abc1f5c-aa9a-4f04-8670-afe09d2a93aa'} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:52.018774 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6abc1f5c-aa9a-4f04-8670-afe09d2a93aa admin admin] The token cache is being invalidate because project bcbe9f2e8f6a4faeb418d1bbc59a47dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81404) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 01:29:52.019140 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6abc1f5c-aa9a-4f04-8670-afe09d2a93aa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:52.019704 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG keystone.notifications [None req-6abc1f5c-aa9a-4f04-8670-afe09d2a93aa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81404) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 01:29:52.021191 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: [pid: 81404|app: 0|req: 1459/2929] 10.222.0.15 () {58 vars in 1233 bytes} [Tue Sep 13 01:29:51 2022] DELETE /identity/v3/projects/bcbe9f2e8f6a4faeb418d1bbc59a47dc => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 01:30:01.666366 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=28,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:30:02.014118 dv-cinder-856941-3-iscsi devstack@keystone.service[81404]: DEBUG dbcounter [-] [81404] Writing DB stats keystone:SELECT=47,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81404) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 01:31:54.128918 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8054c32-3b4a-4417-be8c-51f17dcdf0a4 admin admin] Authenticating user token {{(pid=81405) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 01:31:54.130334 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-f8054c32-3b4a-4417-be8c-51f17dcdf0a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:31:54.199754 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8054c32-3b4a-4417-be8c-51f17dcdf0a4 admin admin] Validating token access rules against request {{(pid=81405) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 01:31:54.257414 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89d7d2f-a950-4e70-b1ae-9bf9c3032866 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8eaff1b843a145a49994dcf089f44315', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e5c1519fdae742f4a25eb7cd5476a8cb', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81405) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 01:31:54.258022 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89d7d2f-a950-4e70-b1ae-9bf9c3032866 service placement] REQUEST_METHOD: `GET` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 01:31:54.258301 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89d7d2f-a950-4e70-b1ae-9bf9c3032866 service placement] SCRIPT_NAME: `/identity` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 01:31:54.258575 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89d7d2f-a950-4e70-b1ae-9bf9c3032866 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81405) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 01:31:54.260309 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.fernet_utils [None req-a89d7d2f-a950-4e70-b1ae-9bf9c3032866 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81405) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 01:31:54.271850 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89d7d2f-a950-4e70-b1ae-9bf9c3032866 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 01:31:54.273399 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89d7d2f-a950-4e70-b1ae-9bf9c3032866 service placement] RBAC: Authorization granted {{(pid=81405) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 01:31:54.283873 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: [pid: 81405|app: 0|req: 1471/2930] 10.222.0.15 () {60 vars in 1365 bytes} [Tue Sep 13 01:31:54 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 01:32:04.278993 dv-cinder-856941-3-iscsi devstack@keystone.service[81405]: DEBUG dbcounter [-] [81405] Writing DB stats keystone:SELECT=58 {{(pid=81405) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}